Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 448.453636ms FAIL (2025-09-23 22:21:44 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 23 22:20:52.637632002 (608h12m27.99041711s) [upload=535247]: Starting upload Sep 23 22:20:52.637680653 (608h12m27.990465762s) [upload=535247]: Done starting upload Sep 23 22:20:52.637743311 (608h12m27.990528409s) [upload=535247]: Uploading segments... Sep 23 22:20:52.637892361 (608h12m27.990677459s) [upload=535247]: Got next segment. Inline: false Sep 23 22:20:52.637920774 (608h12m27.990705872s) [upload=535247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 22:20:52.637924431 (608h12m27.990709539s) [upload=535247]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 22:20:52.658428174 (608h12m28.011213302s) [seg_pos={0 0}, upload=535247]: Begin upload segment... Sep 23 22:20:52.658461807 (608h12m28.011246935s) [seg_pos={0 0}, upload=535247]: Done begin upload segment. Sep 23 22:20:52.658486533 (608h12m28.011271662s) [noise=true , address=45.91.116.8:4048 , piece_id=A2HRBAQMNFSVI72PB72KQ6PP66E264NRF2WXHN7YCCSDMAVHP2BQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.658514636 (608h12m28.011299764s) [noise=true , address=77.90.31.16:4734 , piece_id=HGLU6YK2KNEBVEWLE7FYHIRHKYYYACUHGIDTWSPSOY5ET6DHWMMA, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.658908004 (608h12m28.011693161s) [noise=true , address=45.91.116.14:4189 , piece_id=F7PW7IYZC3JMO6YX6TPEQQ7HOVCLCQPKCQQ7XQZGNLVUEVZG5HUQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.658946516 (608h12m28.011731644s) [noise=true , address=145.79.227.6:4257 , piece_id=5OKZG5K2RFQQJEGUVKQX2N3PVP73K4DPEMFMUIN6L2JDUJSXPWQQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.659342440 (608h12m28.012127567s) [noise=true , address=145.79.227.19:4079 , piece_id=IMUB6DN5ITHQITT37CZYT2PCYJBULMIH2AFCV57WVC26J7JAZR6A, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.659371394 (608h12m28.012156522s) [noise=true , address=45.91.116.10:4081 , piece_id=UD5YNNNHVKDLFNPVX473WDYACWGUDI4NOY2GPN32NKY772LNVRDQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.659693949 (608h12m28.012479076s) [noise=true , address=45.91.116.7:4010 , piece_id=5EECA4MGV2ETEEECG7PSMZ25MPCQDJ5UTZ37GHSNL444J5SFF5IQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.659809786 (608h12m28.012594904s) [noise=true , address=145.79.227.7:4293 , piece_id=ISZW7VW4QRHA7WNMPC2VBNRRADSWGN73NORC2HKRJJK4NBVXDIKQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.660118535 (608h12m28.012903653s) [noise=true , address=145.79.227.15:4874 , piece_id=YKHHORHZS2XCYEBQ5GWUAGQ7IMI7OOR72B6YAM57XJZ3XNZTIDMA, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.660273436 (608h12m28.013058563s) [noise=true , address=45.91.116.10:4082 , piece_id=A5CLRTJ77ZPPY7WKNPQNPC6BZJN52RGLEYY4HAFTPF4IFWX4337A, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.660478080 (608h12m28.013263199s) [noise=true , address=145.79.227.11:4730 , piece_id=BVAYNP4WX65IKBFRHSUYBPQK5W7MKXZZSHJQPMRIPDYUUXAAONLA, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.660644943 (608h12m28.013430061s) [noise=true , address=77.90.31.12:4589 , piece_id=JHNQDY26EU6GFMP2MCQIBVXKQFGPBZEFOYDCGS6UXO5OWMB3UPMQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.660871087 (608h12m28.013656205s) [noise=true , address=77.90.31.13:4627 , piece_id=KIWUPIZ45OGSOCZDA3IXHV5UFF4427S6LK3ERSMHVJ4SZ7HPT4TQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.661011681 (608h12m28.013796809s) [noise=true , address=145.79.227.16:4910 , piece_id=WCZLQ2BTWSAIH45Y7RWXXAPF2AJWASHVPJ6P43QRJJCT5UIFTZFQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.661250409 (608h12m28.014035537s) [noise=true , address=145.79.227.14:4837 , piece_id=C4OJFEVPLL3QQG5M5MGAM3DSTNMGAD6CJLY3B65VRI2DRI5TIQUA, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.661629400 (608h12m28.014414529s) [noise=true , address=77.90.31.14:4663 , piece_id=EGGPZ5M7EJ7ZD2ODUSGDERVSPUOIKHXQYK3YCAUHMSI2XQ54ZGEA, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.773698574 (608h12m28.126483692s) [upload=535247]: Next returned nil segment Sep 23 22:20:52.773818890 (608h12m28.126604008s) [upload=535247]: Waiting for error group managing segments... Sep 23 22:20:52.782263625 (608h12m28.135048733s) [noise=true , address=145.79.227.14:4837 , piece_id=C4OJFEVPLL3QQG5M5MGAM3DSTNMGAD6CJLY3B65VRI2DRI5TIQUA, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.782655320 (608h12m28.135440418s) [noise=true , address=45.91.116.14:4189 , piece_id=F7PW7IYZC3JMO6YX6TPEQQ7HOVCLCQPKCQQ7XQZGNLVUEVZG5HUQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.783325327 (608h12m28.136110436s) [noise=true , address=77.90.31.16:4734 , piece_id=HGLU6YK2KNEBVEWLE7FYHIRHKYYYACUHGIDTWSPSOY5ET6DHWMMA, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.783553115 (608h12m28.136338242s) [noise=true , address=145.79.227.19:4079 , piece_id=IMUB6DN5ITHQITT37CZYT2PCYJBULMIH2AFCV57WVC26J7JAZR6A, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.783664043 (608h12m28.136449151s) [noise=true , address=145.79.227.7:4293 , piece_id=ISZW7VW4QRHA7WNMPC2VBNRRADSWGN73NORC2HKRJJK4NBVXDIKQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.783963324 (608h12m28.136748422s) [noise=true , address=145.79.227.6:4257 , piece_id=5OKZG5K2RFQQJEGUVKQX2N3PVP73K4DPEMFMUIN6L2JDUJSXPWQQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.784735944 (608h12m28.137521042s) [noise=true , address=45.91.116.8:4048 , piece_id=A2HRBAQMNFSVI72PB72KQ6PP66E264NRF2WXHN7YCCSDMAVHP2BQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.785728547 (608h12m28.138513655s) [noise=true , address=77.90.31.13:4627 , piece_id=KIWUPIZ45OGSOCZDA3IXHV5UFF4427S6LK3ERSMHVJ4SZ7HPT4TQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.786062373 (608h12m28.13884747s) [noise=true , address=45.91.116.10:4081 , piece_id=UD5YNNNHVKDLFNPVX473WDYACWGUDI4NOY2GPN32NKY772LNVRDQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.786401659 (608h12m28.139186757s) [noise=true , address=45.91.116.7:4010 , piece_id=5EECA4MGV2ETEEECG7PSMZ25MPCQDJ5UTZ37GHSNL444J5SFF5IQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.786719055 (608h12m28.139504164s) [noise=true , address=145.79.227.11:4730 , piece_id=BVAYNP4WX65IKBFRHSUYBPQK5W7MKXZZSHJQPMRIPDYUUXAAONLA, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.787034076 (608h12m28.139819174s) [noise=true , address=145.79.227.16:4910 , piece_id=WCZLQ2BTWSAIH45Y7RWXXAPF2AJWASHVPJ6P43QRJJCT5UIFTZFQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.787343667 (608h12m28.140128775s) [noise=true , address=77.90.31.12:4589 , piece_id=JHNQDY26EU6GFMP2MCQIBVXKQFGPBZEFOYDCGS6UXO5OWMB3UPMQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.787349117 (608h12m28.140134225s) [seg_pos={0 0}, upload=535247]: Segment reached optimal threshold of 13 pieces. Sep 23 22:20:52.787425150 (608h12m28.140210258s) [noise=true , address=77.90.31.14:4663 , piece_id=EGGPZ5M7EJ7ZD2ODUSGDERVSPUOIKHXQYK3YCAUHMSI2XQ54ZGEA, seg_pos={0 0}, upload=535247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Sep 23 22:20:52.787531679 (608h12m28.140316787s) [noise=true , address=145.79.227.15:4874 , piece_id=YKHHORHZS2XCYEBQ5GWUAGQ7IMI7OOR72B6YAM57XJZ3XNZTIDMA, seg_pos={0 0}, upload=535247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Sep 23 22:20:52.787586312 (608h12m28.14037142s) [noise=true , address=45.91.116.10:4082 , piece_id=A5CLRTJ77ZPPY7WKNPQNPC6BZJN52RGLEYY4HAFTPF4IFWX4337A, seg_pos={0 0}, upload=535247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 23 22:20:52.787595058 (608h12m28.140380166s) [seg_pos={0 0}, upload=535247]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 22:20:52.787636387 (608h12m28.140421485s) [upload=535247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 22:20:52.787639733 (608h12m28.140424841s) [upload=535247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 22:20:52.829148239 (608h12m28.181933377s) [upload=535247]: Done uploading segments... Sep 23 22:20:52.829166583 (608h12m28.181951711s) [upload=535247]: Upload finished. err: <nil> Duration: 448.475687ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002078352s FAIL (2025-09-23 22:21:22 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 23 22:20:52.637632002 (608h12m27.99041711s) [upload=535247]: Starting upload Sep 23 22:20:52.637680653 (608h12m27.990465762s) [upload=535247]: Done starting upload Sep 23 22:20:52.637743311 (608h12m27.990528409s) [upload=535247]: Uploading segments... Sep 23 22:20:52.637892361 (608h12m27.990677459s) [upload=535247]: Got next segment. Inline: false Sep 23 22:20:52.637920774 (608h12m27.990705872s) [upload=535247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 22:20:52.637924431 (608h12m27.990709539s) [upload=535247]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 22:20:52.658428174 (608h12m28.011213302s) [seg_pos={0 0}, upload=535247]: Begin upload segment... Sep 23 22:20:52.658461807 (608h12m28.011246935s) [seg_pos={0 0}, upload=535247]: Done begin upload segment. Sep 23 22:20:52.658486533 (608h12m28.011271662s) [noise=true , address=45.91.116.8:4048 , piece_id=A2HRBAQMNFSVI72PB72KQ6PP66E264NRF2WXHN7YCCSDMAVHP2BQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.658514636 (608h12m28.011299764s) [noise=true , address=77.90.31.16:4734 , piece_id=HGLU6YK2KNEBVEWLE7FYHIRHKYYYACUHGIDTWSPSOY5ET6DHWMMA, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.658908004 (608h12m28.011693161s) [noise=true , address=45.91.116.14:4189 , piece_id=F7PW7IYZC3JMO6YX6TPEQQ7HOVCLCQPKCQQ7XQZGNLVUEVZG5HUQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.658946516 (608h12m28.011731644s) [noise=true , address=145.79.227.6:4257 , piece_id=5OKZG5K2RFQQJEGUVKQX2N3PVP73K4DPEMFMUIN6L2JDUJSXPWQQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.659342440 (608h12m28.012127567s) [noise=true , address=145.79.227.19:4079 , piece_id=IMUB6DN5ITHQITT37CZYT2PCYJBULMIH2AFCV57WVC26J7JAZR6A, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.659371394 (608h12m28.012156522s) [noise=true , address=45.91.116.10:4081 , piece_id=UD5YNNNHVKDLFNPVX473WDYACWGUDI4NOY2GPN32NKY772LNVRDQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.659693949 (608h12m28.012479076s) [noise=true , address=45.91.116.7:4010 , piece_id=5EECA4MGV2ETEEECG7PSMZ25MPCQDJ5UTZ37GHSNL444J5SFF5IQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.659809786 (608h12m28.012594904s) [noise=true , address=145.79.227.7:4293 , piece_id=ISZW7VW4QRHA7WNMPC2VBNRRADSWGN73NORC2HKRJJK4NBVXDIKQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.660118535 (608h12m28.012903653s) [noise=true , address=145.79.227.15:4874 , piece_id=YKHHORHZS2XCYEBQ5GWUAGQ7IMI7OOR72B6YAM57XJZ3XNZTIDMA, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.660273436 (608h12m28.013058563s) [noise=true , address=45.91.116.10:4082 , piece_id=A5CLRTJ77ZPPY7WKNPQNPC6BZJN52RGLEYY4HAFTPF4IFWX4337A, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.660478080 (608h12m28.013263199s) [noise=true , address=145.79.227.11:4730 , piece_id=BVAYNP4WX65IKBFRHSUYBPQK5W7MKXZZSHJQPMRIPDYUUXAAONLA, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.660644943 (608h12m28.013430061s) [noise=true , address=77.90.31.12:4589 , piece_id=JHNQDY26EU6GFMP2MCQIBVXKQFGPBZEFOYDCGS6UXO5OWMB3UPMQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.660871087 (608h12m28.013656205s) [noise=true , address=77.90.31.13:4627 , piece_id=KIWUPIZ45OGSOCZDA3IXHV5UFF4427S6LK3ERSMHVJ4SZ7HPT4TQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.661011681 (608h12m28.013796809s) [noise=true , address=145.79.227.16:4910 , piece_id=WCZLQ2BTWSAIH45Y7RWXXAPF2AJWASHVPJ6P43QRJJCT5UIFTZFQ, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.661250409 (608h12m28.014035537s) [noise=true , address=145.79.227.14:4837 , piece_id=C4OJFEVPLL3QQG5M5MGAM3DSTNMGAD6CJLY3B65VRI2DRI5TIQUA, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.661629400 (608h12m28.014414529s) [noise=true , address=77.90.31.14:4663 , piece_id=EGGPZ5M7EJ7ZD2ODUSGDERVSPUOIKHXQYK3YCAUHMSI2XQ54ZGEA, seg_pos={0 0}, upload=535247]: Uploading piece... Sep 23 22:20:52.773698574 (608h12m28.126483692s) [upload=535247]: Next returned nil segment Sep 23 22:20:52.773818890 (608h12m28.126604008s) [upload=535247]: Waiting for error group managing segments... Sep 23 22:20:52.782263625 (608h12m28.135048733s) [noise=true , address=145.79.227.14:4837 , piece_id=C4OJFEVPLL3QQG5M5MGAM3DSTNMGAD6CJLY3B65VRI2DRI5TIQUA, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.782655320 (608h12m28.135440418s) [noise=true , address=45.91.116.14:4189 , piece_id=F7PW7IYZC3JMO6YX6TPEQQ7HOVCLCQPKCQQ7XQZGNLVUEVZG5HUQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.783325327 (608h12m28.136110436s) [noise=true , address=77.90.31.16:4734 , piece_id=HGLU6YK2KNEBVEWLE7FYHIRHKYYYACUHGIDTWSPSOY5ET6DHWMMA, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.783553115 (608h12m28.136338242s) [noise=true , address=145.79.227.19:4079 , piece_id=IMUB6DN5ITHQITT37CZYT2PCYJBULMIH2AFCV57WVC26J7JAZR6A, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.783664043 (608h12m28.136449151s) [noise=true , address=145.79.227.7:4293 , piece_id=ISZW7VW4QRHA7WNMPC2VBNRRADSWGN73NORC2HKRJJK4NBVXDIKQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.783963324 (608h12m28.136748422s) [noise=true , address=145.79.227.6:4257 , piece_id=5OKZG5K2RFQQJEGUVKQX2N3PVP73K4DPEMFMUIN6L2JDUJSXPWQQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.784735944 (608h12m28.137521042s) [noise=true , address=45.91.116.8:4048 , piece_id=A2HRBAQMNFSVI72PB72KQ6PP66E264NRF2WXHN7YCCSDMAVHP2BQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.785728547 (608h12m28.138513655s) [noise=true , address=77.90.31.13:4627 , piece_id=KIWUPIZ45OGSOCZDA3IXHV5UFF4427S6LK3ERSMHVJ4SZ7HPT4TQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.786062373 (608h12m28.13884747s) [noise=true , address=45.91.116.10:4081 , piece_id=UD5YNNNHVKDLFNPVX473WDYACWGUDI4NOY2GPN32NKY772LNVRDQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.786401659 (608h12m28.139186757s) [noise=true , address=45.91.116.7:4010 , piece_id=5EECA4MGV2ETEEECG7PSMZ25MPCQDJ5UTZ37GHSNL444J5SFF5IQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.786719055 (608h12m28.139504164s) [noise=true , address=145.79.227.11:4730 , piece_id=BVAYNP4WX65IKBFRHSUYBPQK5W7MKXZZSHJQPMRIPDYUUXAAONLA, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.787034076 (608h12m28.139819174s) [noise=true , address=145.79.227.16:4910 , piece_id=WCZLQ2BTWSAIH45Y7RWXXAPF2AJWASHVPJ6P43QRJJCT5UIFTZFQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.787343667 (608h12m28.140128775s) [noise=true , address=77.90.31.12:4589 , piece_id=JHNQDY26EU6GFMP2MCQIBVXKQFGPBZEFOYDCGS6UXO5OWMB3UPMQ, seg_pos={0 0}, upload=535247]: Done uploading piece. err: <nil> Sep 23 22:20:52.787349117 (608h12m28.140134225s) [seg_pos={0 0}, upload=535247]: Segment reached optimal threshold of 13 pieces. Sep 23 22:20:52.787425150 (608h12m28.140210258s) [noise=true , address=77.90.31.14:4663 , piece_id=EGGPZ5M7EJ7ZD2ODUSGDERVSPUOIKHXQYK3YCAUHMSI2XQ54ZGEA, seg_pos={0 0}, upload=535247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Sep 23 22:20:52.787531679 (608h12m28.140316787s) [noise=true , address=145.79.227.15:4874 , piece_id=YKHHORHZS2XCYEBQ5GWUAGQ7IMI7OOR72B6YAM57XJZ3XNZTIDMA, seg_pos={0 0}, upload=535247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Sep 23 22:20:52.787586312 (608h12m28.14037142s) [noise=true , address=45.91.116.10:4082 , piece_id=A5CLRTJ77ZPPY7WKNPQNPC6BZJN52RGLEYY4HAFTPF4IFWX4337A, seg_pos={0 0}, upload=535247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 23 22:20:52.787595058 (608h12m28.140380166s) [seg_pos={0 0}, upload=535247]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 22:20:52.787636387 (608h12m28.140421485s) [upload=535247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 22:20:52.787639733 (608h12m28.140424841s) [upload=535247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 22:20:52.829148239 (608h12m28.181933377s) [upload=535247]: Done uploading segments... Sep 23 22:20:52.829166583 (608h12m28.181951711s) [upload=535247]: Upload finished. err: <nil> Duration: 30.002102636s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 439.793775ms FAIL (2025-09-23 19:36:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 439.815947ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 423.047525ms FAIL (2025-09-23 19:35:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 423.068803ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 558.983453ms FAIL (2025-09-23 19:34:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 559.170584ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 397.620413ms FAIL (2025-09-23 19:34:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 397.643555ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 387.962323ms FAIL (2025-09-23 19:34:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 387.992631ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.XXX........................................................] duration: 652.809799ms FAIL (2025-09-23 19:34:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 652.846349ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 451.18606ms FAIL (2025-09-23 19:33:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 451.208693ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration: 520.159046ms FAIL (2025-09-23 19:33:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 520.181709ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.X........................................................] duration: 332.759411ms FAIL (2025-09-23 19:33:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 332.795749ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...XXX......................................................] duration: 433.331916ms FAIL (2025-09-23 19:32:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 433.350652ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.X.......................................................] duration: 534.747765ms FAIL (2025-09-23 19:32:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 534.816505ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration: 411.220566ms FAIL (2025-09-23 19:31:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 411.26559ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.X......................................................] duration: 427.486563ms FAIL (2025-09-23 19:30:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.509376ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.X.....................................................] duration: 440.768292ms FAIL (2025-09-23 19:30:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 440.783851ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....XXX....................................................] duration: 394.868685ms FAIL (2025-09-23 19:30:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 394.894042ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.X....................................................] duration: 421.494532ms FAIL (2025-09-23 19:29:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 421.521904ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.X...................................................] duration: 396.148113ms FAIL (2025-09-23 19:29:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 396.175014ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......XXX...................................................] duration: 427.917184ms FAIL (2025-09-23 19:29:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 427.938214ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.X..................................................] duration: 342.664634ms FAIL (2025-09-23 19:28:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 342.692595ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......XXX..................................................] duration: 434.762941ms FAIL (2025-09-23 19:28:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 434.791105ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.X.................................................] duration: 394.563483ms FAIL (2025-09-23 19:27:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 394.590674ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........XXX.................................................] duration: 455.513946ms FAIL (2025-09-23 19:27:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 455.533483ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.X................................................] duration: 430.744415ms FAIL (2025-09-23 19:27:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 430.766537ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.X...............................................] duration: 370.51681ms FAIL (2025-09-23 19:26:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 370.534342ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.X..............................................] duration: 320.85875ms FAIL (2025-09-23 19:26:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 320.879238ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........XXX................................................] duration: 389.428058ms FAIL (2025-09-23 19:26:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 389.446152ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........XXX...............................................] duration: 460.963341ms FAIL (2025-09-23 19:26:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 460.98907ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........XXX..............................................] duration: 677.896708ms FAIL (2025-09-23 19:25:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 677.91413ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.X.............................................] duration: 441.317616ms FAIL (2025-09-23 19:25:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.341441ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............XXX.............................................] duration: 417.931423ms FAIL (2025-09-23 19:25:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 417.949268ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.X............................................] duration: 355.042815ms FAIL (2025-09-23 19:24:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 355.072711ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.X...........................................] duration: 456.915581ms FAIL (2025-09-23 19:24:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.941639ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............XXX............................................] duration: 393.836332ms FAIL (2025-09-23 19:23:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 393.856029ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............XXX...........................................] duration: 425.430829ms FAIL (2025-09-23 19:23:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 425.456838ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration: 523.582015ms FAIL (2025-09-23 19:22:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 523.679209ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............XXX..........................................] duration: 512.41394ms FAIL (2025-09-23 19:22:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 512.452463ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.X.........................................] duration: 423.449285ms FAIL (2025-09-23 19:21:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 423.471015ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................XXX.........................................] duration: 393.816893ms FAIL (2025-09-23 19:21:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 393.836109ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.X........................................] duration: 366.627221ms FAIL (2025-09-23 19:20:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 366.653049ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.X.......................................] duration: 636.231844ms FAIL (2025-09-23 19:20:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 636.255299ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................XXX........................................] duration: 420.795775ms FAIL (2025-09-23 19:20:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 420.813888ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.X......................................] duration: 472.391567ms FAIL (2025-09-23 19:19:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 472.415351ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................XXX.......................................] duration: 827.398263ms FAIL (2025-09-23 19:19:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 827.422087ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 419.884405ms FAIL (2025-09-23 19:18:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 419.912347ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................XXX......................................] duration: 453.875351ms FAIL (2025-09-23 19:18:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 453.897933ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 411.581523ms FAIL (2025-09-23 19:18:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 411.601781ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................XXX.....................................] duration: 393.596529ms FAIL (2025-09-23 19:18:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 393.615224ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.X...................................] duration: 428.306803ms FAIL (2025-09-23 19:17:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 18:50:01.287695314 (604h41m36.640480412s) [upload=532124]: Starting upload Sep 23 18:50:01.287781777 (604h41m36.640566885s) [upload=532124]: Done starting upload Sep 23 18:50:01.287890500 (604h41m36.640675598s) [upload=532124]: Uploading segments... Sep 23 18:50:01.288071600 (604h41m36.640856698s) [upload=532124]: Got next segment. Inline: false Sep 23 18:50:01.288106455 (604h41m36.640891563s) [upload=532124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:50:01.288110964 (604h41m36.640896072s) [upload=532124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:50:01.310802305 (604h41m36.663587413s) [seg_pos={0 0}, upload=532124]: Begin upload segment... Sep 23 18:50:01.310845706 (604h41m36.663630814s) [seg_pos={0 0}, upload=532124]: Done begin upload segment. Sep 23 18:50:01.310866836 (604h41m36.663651934s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.310898745 (604h41m36.663683853s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311189080 (604h41m36.663974178s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311326897 (604h41m36.664111996s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311437104 (604h41m36.664222202s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311683225 (604h41m36.664468324s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311701891 (604h41m36.664486999s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.311958352 (604h41m36.66474346s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312034805 (604h41m36.664819903s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312192901 (604h41m36.664978s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312315822 (604h41m36.665100931s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312476534 (604h41m36.665261632s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312589385 (604h41m36.665374493s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312794240 (604h41m36.665579337s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.312839294 (604h41m36.665624392s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.313085636 (604h41m36.665870744s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Uploading piece... Sep 23 18:50:01.383001332 (604h41m36.735786429s) [upload=532124]: Next returned nil segment Sep 23 18:50:01.383018864 (604h41m36.735803963s) [upload=532124]: Waiting for error group managing segments... Sep 23 18:50:01.393293842 (604h41m36.746078941s) [noise=true , address=45.91.116.4:4939 , piece_id=YHHNEK2MLYSMGKQIZFYK5KXJBILUZCAYS33CKPRMMYTEMPAAV76Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393496663 (604h41m36.746281771s) [noise=true , address=45.91.116.14:4191 , piece_id=ODIHVEIXNGCJZL26DCCBUWDXBSGRFKSRROXSUDPRKWYYSXN6QZKQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.393714611 (604h41m36.746499709s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7OWOKUAIK6JLDH7YVVYF4CJJJLAJAJWPN35ZJZ226H7FWNTSUHQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395029667 (604h41m36.747814765s) [noise=true , address=77.90.31.10:4519 , piece_id=Y3CMSIGU32ZZKHBHZXKNJHYRNRW5XRHWIIS2PWZUFOKRTJRXFPWA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.395099057 (604h41m36.747884165s) [noise=true , address=77.90.31.13:4627 , piece_id=73P42LODXHBCLZSKBRJXLZWKFLKSHQ52JFUX34MUSKB4P7SI3YNQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.396395348 (604h41m36.749180446s) [noise=true , address=77.90.31.21:4877 , piece_id=R5FQYKR6UXTU2HZIMYJK7UATRBXLMFYCURCMZUUKFS3ZDKHSFLUQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:01.397286269 (604h41m36.750071357s) [noise=true , address=77.90.31.19:4841 , piece_id=FPJ2NH2CHB66PZN3QAZKGGH6HFFP3XBMQE7CD66CM3Q7JG2OTEEA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:06.503336259 (604h41m41.856121367s) [noise=true , address=145.79.227.14:4840 , piece_id=VIC5QXRQZK7RW5PIJWEJKZ75RHMPO6E3KQU3GSPJ7QXC4Z3YGWMQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.622047002 (604h41m42.97483211s) [noise=true , address=145.79.227.7:4294 , piece_id=UC6OPR6O6LAT67F5QASBQEKTQDCRNU46KZSRTRGITMIAQ3DMCHNA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:07.636026405 (604h41m42.988811503s) [noise=true , address=145.79.227.4:4187 , piece_id=QRQE2SMZJYXNOPSHT5R7LQ5QE7UZBYQGXUMUVQUIWXXNSMXALF4Q, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:18.247630360 (604h41m53.600415449s) [noise=true , address=145.79.227.19:4077 , piece_id=IQ267NE2U4NBUK67G2H2H3RBWESULCZT7OWMSGOWXNF6NDZ2X3YA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:26.526048179 (604h42m1.878833277s) [noise=true , address=145.79.227.16:4909 , piece_id=KR63YT75RZAPNK43TMM6DX6PAKIZZKBVIPDYYWCTGKPZX566WJIQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: <nil> Sep 23 18:50:31.276564528 (604h42m6.629349647s) [noise=true , address=145.79.227.15:4876 , piece_id=7PKRZ5B7DLSWZRMMYAXRZMXZCJRFJX6QB7OEJKJQ3X4D7ESFSOGA, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276644659 (604h42m6.629429777s) [noise=true , address=145.79.227.13:4804 , piece_id=U6LUHJ2JC3H4UXX7PFNJOWA2BTMW7YYKDEFGDN7TYX4VS7I22IRQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:50:31.276704250 (604h42m6.629489378s) [noise=true , address=145.79.227.6:4258 , piece_id=I5QTAEG7ZLBGYZLAMPFQ5MQSVEBYK54D4FIQ2DH6JZ65YNEXSHYQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:53926->145.79.227.6:4258: use of closed network connection Sep 23 18:50:31.276782718 (604h42m6.629567845s) [noise=true , address=145.79.227.18:4005 , piece_id=RZJQV5CZ7LFKCWBXWREC2XR5VXVKQ22HOOWOGNGN7SVK4XQOPJBQ, seg_pos={0 0}, upload=532124]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 5.161.89.252:59332->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 23 18:50:31.276828955 (604h42m6.629614082s) [seg_pos={0 0}, upload=532124]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 18:50:31.290415482 (604h42m6.64320062s) [upload=532124]: Done uploading segments... Sep 23 18:50:31.290453714 (604h42m6.643238882s) [upload=532124]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 428.334285ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)