Orbiter History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.063189407s FAIL (2025-09-22 03:42:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.06321263s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [X.......XX..................................................] duration: 2.240454704s FAIL (2025-09-22 03:42:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.240479621s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X....................................X..X...X...X.X.....XX.] duration: 715.509473ms FAIL (2025-09-22 03:42:24 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:41:53.821360598 (565h33m29.174145726s) [upload=497735]: Starting upload Sep 22 03:41:53.821432342 (565h33m29.174217471s) [upload=497735]: Done starting upload Sep 22 03:41:53.821573697 (565h33m29.174358826s) [upload=497735]: Uploading segments... Sep 22 03:41:53.821895693 (565h33m29.174680821s) [upload=497735]: Got next segment. Inline: false Sep 22 03:41:53.821989369 (565h33m29.174774496s) [upload=497735]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:41:53.821995941 (565h33m29.174781069s) [upload=497735]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:41:53.843161018 (565h33m29.195946156s) [seg_pos={0 0}, upload=497735]: Begin upload segment... Sep 22 03:41:53.843219267 (565h33m29.196004405s) [seg_pos={0 0}, upload=497735]: Done begin upload segment. Sep 22 03:41:53.843260845 (565h33m29.196045973s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843270974 (565h33m29.196056102s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843790710 (565h33m29.196575838s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844019850 (565h33m29.196804988s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844360168 (565h33m29.197145296s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844472439 (565h33m29.197257576s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844820201 (565h33m29.19760533s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844922043 (565h33m29.197707171s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845271628 (565h33m29.198056767s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845900639 (565h33m29.198685767s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846045771 (565h33m29.1988309s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846725577 (565h33m29.199510705s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847449616 (565h33m29.200234744s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847961646 (565h33m29.200746775s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.849104080 (565h33m29.201889208s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.850005882 (565h33m29.202791011s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.982635031 (565h33m29.335420168s) [upload=497735]: Next returned nil segment Sep 22 03:41:53.982652824 (565h33m29.335437952s) [upload=497735]: Waiting for error group managing segments... Sep 22 03:41:53.990044095 (565h33m29.342829232s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.990531790 (565h33m29.343316918s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991047046 (565h33m29.343832184s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991466263 (565h33m29.344251392s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991769693 (565h33m29.344554831s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992458005 (565h33m29.345243133s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992505003 (565h33m29.345290131s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993159922 (565h33m29.34594505s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993274718 (565h33m29.346059845s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994692838 (565h33m29.347477966s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994841236 (565h33m29.347626365s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.995361182 (565h33m29.348146311s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:42:23.809279587 (565h33m59.162064714s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809421593 (565h33m59.162206721s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809560264 (565h33m59.162345402s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:42:23.809598205 (565h33m59.162383333s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; context deadline exceeded Sep 22 03:42:23.809711137 (565h33m59.162496265s) [seg_pos={0 0}, upload=497735]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:42:23.825581299 (565h33m59.178366426s) [upload=497735]: Done uploading segments... Sep 22 03:42:23.825600274 (565h33m59.178385412s) [upload=497735]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 715.541174ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X....................................X..X...X...X.X.....XX] duration: 30.017879052s FAIL (2025-09-22 03:42:23 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:41:53.821360598 (565h33m29.174145726s) [upload=497735]: Starting upload Sep 22 03:41:53.821432342 (565h33m29.174217471s) [upload=497735]: Done starting upload Sep 22 03:41:53.821573697 (565h33m29.174358826s) [upload=497735]: Uploading segments... Sep 22 03:41:53.821895693 (565h33m29.174680821s) [upload=497735]: Got next segment. Inline: false Sep 22 03:41:53.821989369 (565h33m29.174774496s) [upload=497735]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:41:53.821995941 (565h33m29.174781069s) [upload=497735]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:41:53.843161018 (565h33m29.195946156s) [seg_pos={0 0}, upload=497735]: Begin upload segment... Sep 22 03:41:53.843219267 (565h33m29.196004405s) [seg_pos={0 0}, upload=497735]: Done begin upload segment. Sep 22 03:41:53.843260845 (565h33m29.196045973s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843270974 (565h33m29.196056102s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843790710 (565h33m29.196575838s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844019850 (565h33m29.196804988s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844360168 (565h33m29.197145296s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844472439 (565h33m29.197257576s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844820201 (565h33m29.19760533s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844922043 (565h33m29.197707171s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845271628 (565h33m29.198056767s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845900639 (565h33m29.198685767s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846045771 (565h33m29.1988309s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846725577 (565h33m29.199510705s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847449616 (565h33m29.200234744s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847961646 (565h33m29.200746775s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.849104080 (565h33m29.201889208s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.850005882 (565h33m29.202791011s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.982635031 (565h33m29.335420168s) [upload=497735]: Next returned nil segment Sep 22 03:41:53.982652824 (565h33m29.335437952s) [upload=497735]: Waiting for error group managing segments... Sep 22 03:41:53.990044095 (565h33m29.342829232s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.990531790 (565h33m29.343316918s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991047046 (565h33m29.343832184s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991466263 (565h33m29.344251392s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991769693 (565h33m29.344554831s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992458005 (565h33m29.345243133s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992505003 (565h33m29.345290131s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993159922 (565h33m29.34594505s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993274718 (565h33m29.346059845s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994692838 (565h33m29.347477966s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994841236 (565h33m29.347626365s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.995361182 (565h33m29.348146311s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:42:23.809279587 (565h33m59.162064714s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809421593 (565h33m59.162206721s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809560264 (565h33m59.162345402s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:42:23.809598205 (565h33m59.162383333s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; context deadline exceeded Sep 22 03:42:23.809711137 (565h33m59.162496265s) [seg_pos={0 0}, upload=497735]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:42:23.825581299 (565h33m59.178366426s) [upload=497735]: Done uploading segments... Sep 22 03:42:23.825600274 (565h33m59.178385412s) [upload=497735]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.017901505s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X.......XX.................................................] duration: 2.200208831s FAIL (2025-09-22 03:42:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.200232095s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..........XX.....X..X...X..........X...X.XX...XXX...X......] duration: 461.118411ms FAIL (2025-09-22 03:42:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 461.146985ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX..........XX.....X..X...X..........X...X.XX...XXX...X.....] duration: 469.077607ms FAIL (2025-09-22 03:41:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 469.136467ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 4.938176841s FAIL (2025-09-22 03:41:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.938205214s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X....................................X..X...X...X.X.....X] duration: 30.015273347s FAIL (2025-09-22 03:41:31 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:41:01.871600412 (565h32m37.22438552s) [upload=497720]: Starting upload Sep 22 03:41:01.871653101 (565h32m37.224438199s) [upload=497720]: Done starting upload Sep 22 03:41:01.871730476 (565h32m37.224515574s) [upload=497720]: Uploading segments... Sep 22 03:41:01.871888362 (565h32m37.22467347s) [upload=497720]: Got next segment. Inline: false Sep 22 03:41:01.871934830 (565h32m37.224719938s) [upload=497720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:41:01.871938877 (565h32m37.224723975s) [upload=497720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:41:01.890319909 (565h32m37.243105018s) [seg_pos={0 0}, upload=497720]: Begin upload segment... Sep 22 03:41:01.890351910 (565h32m37.243137018s) [seg_pos={0 0}, upload=497720]: Done begin upload segment. Sep 22 03:41:01.890379401 (565h32m37.2431645s) [noise=true , address=77.90.31.17:4771 , piece_id=EFBGIIKAC6T5JJOQT5OECSZDXGGGHHIKD3PIOWZV75BIBWC2LRNQ, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.890385283 (565h32m37.24317038s) [noise=true , address=145.79.227.7:4293 , piece_id=XOBJLHJYT3NUY42QZAOWNXHOXTXPWOPTGSAFSIVEICVP4OBN36HA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.890683673 (565h32m37.243468781s) [noise=true , address=77.90.31.19:4842 , piece_id=BAIBGNJQXFGHVLGE4TGQ4SUDZMDBZ4HN7OJHUFDE7VLTDYLNKIEA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.890723799 (565h32m37.243508907s) [noise=true , address=45.91.116.9:4261 , piece_id=7TIM2TTXRIYHWNJWURKPN4NOX2YLUDC33EDERS5D5BI5EEXLKAZA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.890914015 (565h32m37.243699123s) [noise=true , address=45.91.116.10:4081 , piece_id=VATBBU2K7EJSXK5KVP3UV466IBU6R46YXXYW7EUAS4YCHASVSPZQ, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891017149 (565h32m37.243802246s) [noise=true , address=45.91.116.12:4227 , piece_id=7G6UIWRVEOZBROBOOYCZO3PKLNDRLQANDUH464WRNWOBFMWYF22Q, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891160618 (565h32m37.243945715s) [noise=true , address=77.90.31.13:4628 , piece_id=5EIQUAB7OOAT7THIOSZTP5Q624P2R7PT6XDHWMDBOQWDPAWVBR7Q, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891242742 (565h32m37.24402785s) [noise=true , address=145.79.227.6:4257 , piece_id=HP5DMAP6KXCTAT62YMFU6YQGU7RD2QOIBYG3OKR3C67WTKXPRRDA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891397312 (565h32m37.24418242s) [noise=true , address=145.79.227.2:4115 , piece_id=B3PL7B2XB36MNSCAHYEKRA5XHIQRBTNKOR3YPMTWOOQT2DCSKOFA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891471482 (565h32m37.244256579s) [noise=true , address=77.90.31.19:4843 , piece_id=XDZ53A3BYUO3AQOQIXI2MPOIJJND44TP45WQ2G5Q2NZX4DG7TCGA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891720729 (565h32m37.244505837s) [noise=true , address=45.91.116.6:4973 , piece_id=D2GYHIMWS2Z3UX6Y2KVH2AFDI4L3ONKC5PIAGZJH77ZKLSM3B7XQ, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891981528 (565h32m37.244766636s) [noise=true , address=77.90.31.14:4662 , piece_id=LJDZOVSBBTQDIKMWQKDNZWF4PJ3D3V6OZ75DEC5CAKDRFFXLZGSA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.892226558 (565h32m37.245011656s) [noise=true , address=45.91.116.7:4010 , piece_id=N2ORNKABODGSLOAK3MTYRZ7SBZMJUTNVQWHNUBI4OMVTMYRWDUDQ, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.892444628 (565h32m37.245229736s) [noise=true , address=145.79.227.11:4730 , piece_id=OHYMTLDU7HT565ITPYSAWIW6TNGO337QUATZHGKPGG4YPFLOI4IQ, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.894227192 (565h32m37.2470123s) [noise=true , address=45.91.116.11:4117 , piece_id=4DJ55PYUAYQAJIWE5ACBRIU3SXB3UGTDNWZ2FWIYH5G45MRY4LSQ, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.894682617 (565h32m37.247467715s) [noise=true , address=45.91.116.5:4297 , piece_id=7CLUHX2RGSZV6BVFOIFTMKJ3QRKRA3G47KAJYD6PGTUX3WPC2GAA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.971552279 (565h32m37.324337376s) [upload=497720]: Next returned nil segment Sep 22 03:41:01.971565003 (565h32m37.324350111s) [upload=497720]: Waiting for error group managing segments... Sep 22 03:41:01.979626992 (565h32m37.3324121s) [noise=true , address=45.91.116.12:4227 , piece_id=7G6UIWRVEOZBROBOOYCZO3PKLNDRLQANDUH464WRNWOBFMWYF22Q, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.980663607 (565h32m37.333448705s) [noise=true , address=45.91.116.10:4081 , piece_id=VATBBU2K7EJSXK5KVP3UV466IBU6R46YXXYW7EUAS4YCHASVSPZQ, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.981025155 (565h32m37.333810263s) [noise=true , address=45.91.116.7:4010 , piece_id=N2ORNKABODGSLOAK3MTYRZ7SBZMJUTNVQWHNUBI4OMVTMYRWDUDQ, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.981356817 (565h32m37.334141925s) [noise=true , address=45.91.116.5:4297 , piece_id=7CLUHX2RGSZV6BVFOIFTMKJ3QRKRA3G47KAJYD6PGTUX3WPC2GAA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.981713808 (565h32m37.334498916s) [noise=true , address=45.91.116.11:4117 , piece_id=4DJ55PYUAYQAJIWE5ACBRIU3SXB3UGTDNWZ2FWIYH5G45MRY4LSQ, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.982296310 (565h32m37.335081418s) [noise=true , address=45.91.116.9:4261 , piece_id=7TIM2TTXRIYHWNJWURKPN4NOX2YLUDC33EDERS5D5BI5EEXLKAZA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.982611473 (565h32m37.335396581s) [noise=true , address=77.90.31.19:4842 , piece_id=BAIBGNJQXFGHVLGE4TGQ4SUDZMDBZ4HN7OJHUFDE7VLTDYLNKIEA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.983091703 (565h32m37.335876811s) [noise=true , address=77.90.31.17:4771 , piece_id=EFBGIIKAC6T5JJOQT5OECSZDXGGGHHIKD3PIOWZV75BIBWC2LRNQ, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.983409069 (565h32m37.336194177s) [noise=true , address=77.90.31.13:4628 , piece_id=5EIQUAB7OOAT7THIOSZTP5Q624P2R7PT6XDHWMDBOQWDPAWVBR7Q, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.983763253 (565h32m37.336548361s) [noise=true , address=77.90.31.19:4843 , piece_id=XDZ53A3BYUO3AQOQIXI2MPOIJJND44TP45WQ2G5Q2NZX4DG7TCGA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.984167782 (565h32m37.33695289s) [noise=true , address=45.91.116.6:4973 , piece_id=D2GYHIMWS2Z3UX6Y2KVH2AFDI4L3ONKC5PIAGZJH77ZKLSM3B7XQ, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.984372727 (565h32m37.337157835s) [noise=true , address=77.90.31.14:4662 , piece_id=LJDZOVSBBTQDIKMWQKDNZWF4PJ3D3V6OZ75DEC5CAKDRFFXLZGSA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:31.860976990 (565h33m7.213762129s) [noise=true , address=145.79.227.7:4293 , piece_id=XOBJLHJYT3NUY42QZAOWNXHOXTXPWOPTGSAFSIVEICVP4OBN36HA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:41:31.861026934 (565h33m7.213812062s) [noise=true , address=145.79.227.11:4730 , piece_id=OHYMTLDU7HT565ITPYSAWIW6TNGO337QUATZHGKPGG4YPFLOI4IQ, seg_pos={0 0}, upload=497720]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:41:31.861036973 (565h33m7.213822101s) [noise=true , address=145.79.227.2:4115 , piece_id=B3PL7B2XB36MNSCAHYEKRA5XHIQRBTNKOR3YPMTWOOQT2DCSKOFA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:41:31.861047242 (565h33m7.21383237s) [noise=true , address=145.79.227.6:4257 , piece_id=HP5DMAP6KXCTAT62YMFU6YQGU7RD2QOIBYG3OKR3C67WTKXPRRDA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:41:31.861102175 (565h33m7.213887304s) [seg_pos={0 0}, upload=497720]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:41:31.873967859 (565h33m7.226752987s) [upload=497720]: Done uploading segments... Sep 22 03:41:31.873983789 (565h33m7.226768917s) [upload=497720]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.015287193s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X.......XX................................................] duration: 2.18619079s FAIL (2025-09-22 03:41:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.18621716s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.306684651s FAIL (2025-09-22 03:41:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.306706332s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX..........XX.....X..X...X..........X...X.XX...XXX...X....] duration: 705.231178ms FAIL (2025-09-22 03:40:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 705.253159ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.054300696s FAIL (2025-09-22 03:40:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.054352113s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...X.......XX...............................................] duration: 2.158866688s FAIL (2025-09-22 03:40:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.158891144s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X....................................X..X...X...X.X.....] duration: 378.487169ms FAIL (2025-09-22 03:40:30 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:37:05.953774132 (565h28m41.30655924s) [upload=497661]: Starting upload Sep 22 03:37:05.953823264 (565h28m41.306608372s) [upload=497661]: Done starting upload Sep 22 03:37:05.953885071 (565h28m41.306670168s) [upload=497661]: Uploading segments... Sep 22 03:37:05.954033739 (565h28m41.306818837s) [upload=497661]: Got next segment. Inline: false Sep 22 03:37:05.954058936 (565h28m41.306844044s) [upload=497661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:37:05.954062213 (565h28m41.306847321s) [upload=497661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:37:05.978394584 (565h28m41.331179682s) [seg_pos={0 0}, upload=497661]: Begin upload segment... Sep 22 03:37:05.978424721 (565h28m41.331209829s) [seg_pos={0 0}, upload=497661]: Done begin upload segment. Sep 22 03:37:05.978443977 (565h28m41.331229085s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978484633 (565h28m41.331269741s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978762044 (565h28m41.331547153s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978783665 (565h28m41.331568773s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979021161 (565h28m41.331806269s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979040176 (565h28m41.331825274s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979239009 (565h28m41.332024117s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979346039 (565h28m41.332131147s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979977655 (565h28m41.332762763s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980195954 (565h28m41.332981053s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980470450 (565h28m41.333255558s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980850994 (565h28m41.333636092s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981390416 (565h28m41.334175524s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981549866 (565h28m41.334334974s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982114485 (565h28m41.334899593s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982508565 (565h28m41.335293663s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:06.088031568 (565h28m41.440816685s) [upload=497661]: Next returned nil segment Sep 22 03:37:06.088047888 (565h28m41.440832996s) [upload=497661]: Waiting for error group managing segments... Sep 22 03:37:06.097235449 (565h28m41.450020588s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098300347 (565h28m41.451085445s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098716739 (565h28m41.451501847s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098985253 (565h28m41.451770381s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099078658 (565h28m41.451863767s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099415269 (565h28m41.452200377s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099811354 (565h28m41.452596462s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099992072 (565h28m41.452777201s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100240468 (565h28m41.453025576s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100627725 (565h28m41.453412833s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100834132 (565h28m41.453619271s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.101022145 (565h28m41.453807254s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:35.942202163 (565h29m11.294987262s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942247639 (565h29m11.295032737s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942299857 (565h29m11.295084985s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942350993 (565h29m11.295136091s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942376942 (565h29m11.29516205s) [seg_pos={0 0}, upload=497661]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:37:35.957725684 (565h29m11.310510792s) [upload=497661]: Done uploading segments... Sep 22 03:37:35.957737848 (565h29m11.310522956s) [upload=497661]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 378.508199ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.155591917s FAIL (2025-09-22 03:40:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.155609259s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X....................................X..X...X...X.X....] duration: 522.529736ms FAIL (2025-09-22 03:40:05 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:37:05.953774132 (565h28m41.30655924s) [upload=497661]: Starting upload Sep 22 03:37:05.953823264 (565h28m41.306608372s) [upload=497661]: Done starting upload Sep 22 03:37:05.953885071 (565h28m41.306670168s) [upload=497661]: Uploading segments... Sep 22 03:37:05.954033739 (565h28m41.306818837s) [upload=497661]: Got next segment. Inline: false Sep 22 03:37:05.954058936 (565h28m41.306844044s) [upload=497661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:37:05.954062213 (565h28m41.306847321s) [upload=497661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:37:05.978394584 (565h28m41.331179682s) [seg_pos={0 0}, upload=497661]: Begin upload segment... Sep 22 03:37:05.978424721 (565h28m41.331209829s) [seg_pos={0 0}, upload=497661]: Done begin upload segment. Sep 22 03:37:05.978443977 (565h28m41.331229085s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978484633 (565h28m41.331269741s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978762044 (565h28m41.331547153s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978783665 (565h28m41.331568773s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979021161 (565h28m41.331806269s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979040176 (565h28m41.331825274s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979239009 (565h28m41.332024117s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979346039 (565h28m41.332131147s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979977655 (565h28m41.332762763s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980195954 (565h28m41.332981053s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980470450 (565h28m41.333255558s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980850994 (565h28m41.333636092s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981390416 (565h28m41.334175524s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981549866 (565h28m41.334334974s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982114485 (565h28m41.334899593s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982508565 (565h28m41.335293663s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:06.088031568 (565h28m41.440816685s) [upload=497661]: Next returned nil segment Sep 22 03:37:06.088047888 (565h28m41.440832996s) [upload=497661]: Waiting for error group managing segments... Sep 22 03:37:06.097235449 (565h28m41.450020588s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098300347 (565h28m41.451085445s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098716739 (565h28m41.451501847s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098985253 (565h28m41.451770381s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099078658 (565h28m41.451863767s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099415269 (565h28m41.452200377s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099811354 (565h28m41.452596462s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099992072 (565h28m41.452777201s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100240468 (565h28m41.453025576s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100627725 (565h28m41.453412833s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100834132 (565h28m41.453619271s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.101022145 (565h28m41.453807254s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:35.942202163 (565h29m11.294987262s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942247639 (565h29m11.295032737s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942299857 (565h29m11.295084985s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942350993 (565h29m11.295136091s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942376942 (565h29m11.29516205s) [seg_pos={0 0}, upload=497661]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:37:35.957725684 (565h29m11.310510792s) [upload=497661]: Done uploading segments... Sep 22 03:37:35.957737848 (565h29m11.310522956s) [upload=497661]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 522.560575ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X.......XX..............................................] duration: 2.313443937s FAIL (2025-09-22 03:39:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.313470266s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..XX..........XX.....X..X...X..........X...X.XX...XXX...X...] duration: 423.72487ms FAIL (2025-09-22 03:39:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 423.779432ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.007407117s FAIL (2025-09-22 03:39:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.007427365s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X....................................X..X...X...X.X...] duration: 400.93749ms FAIL (2025-09-22 03:39:17 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:37:05.953774132 (565h28m41.30655924s) [upload=497661]: Starting upload Sep 22 03:37:05.953823264 (565h28m41.306608372s) [upload=497661]: Done starting upload Sep 22 03:37:05.953885071 (565h28m41.306670168s) [upload=497661]: Uploading segments... Sep 22 03:37:05.954033739 (565h28m41.306818837s) [upload=497661]: Got next segment. Inline: false Sep 22 03:37:05.954058936 (565h28m41.306844044s) [upload=497661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:37:05.954062213 (565h28m41.306847321s) [upload=497661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:37:05.978394584 (565h28m41.331179682s) [seg_pos={0 0}, upload=497661]: Begin upload segment... Sep 22 03:37:05.978424721 (565h28m41.331209829s) [seg_pos={0 0}, upload=497661]: Done begin upload segment. Sep 22 03:37:05.978443977 (565h28m41.331229085s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978484633 (565h28m41.331269741s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978762044 (565h28m41.331547153s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978783665 (565h28m41.331568773s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979021161 (565h28m41.331806269s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979040176 (565h28m41.331825274s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979239009 (565h28m41.332024117s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979346039 (565h28m41.332131147s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979977655 (565h28m41.332762763s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980195954 (565h28m41.332981053s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980470450 (565h28m41.333255558s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980850994 (565h28m41.333636092s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981390416 (565h28m41.334175524s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981549866 (565h28m41.334334974s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982114485 (565h28m41.334899593s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982508565 (565h28m41.335293663s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:06.088031568 (565h28m41.440816685s) [upload=497661]: Next returned nil segment Sep 22 03:37:06.088047888 (565h28m41.440832996s) [upload=497661]: Waiting for error group managing segments... Sep 22 03:37:06.097235449 (565h28m41.450020588s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098300347 (565h28m41.451085445s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098716739 (565h28m41.451501847s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098985253 (565h28m41.451770381s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099078658 (565h28m41.451863767s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099415269 (565h28m41.452200377s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099811354 (565h28m41.452596462s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099992072 (565h28m41.452777201s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100240468 (565h28m41.453025576s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100627725 (565h28m41.453412833s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100834132 (565h28m41.453619271s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.101022145 (565h28m41.453807254s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:35.942202163 (565h29m11.294987262s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942247639 (565h29m11.295032737s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942299857 (565h29m11.295084985s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942350993 (565h29m11.295136091s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942376942 (565h29m11.29516205s) [seg_pos={0 0}, upload=497661]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:37:35.957725684 (565h29m11.310510792s) [upload=497661]: Done uploading segments... Sep 22 03:37:35.957737848 (565h29m11.310522956s) [upload=497661]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 400.955936ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...XX..........XX.....X..X...X..........X...X.XX...XXX...X..] duration: 433.867664ms FAIL (2025-09-22 03:39:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 433.884045ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 4.908807947s FAIL (2025-09-22 03:39:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.908831151s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......XX.............................................] duration: 2.188699483s FAIL (2025-09-22 03:38:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.188718638s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X....................................X..X...X...X.X..] duration: 473.50099ms FAIL (2025-09-22 03:38:45 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:37:05.953774132 (565h28m41.30655924s) [upload=497661]: Starting upload Sep 22 03:37:05.953823264 (565h28m41.306608372s) [upload=497661]: Done starting upload Sep 22 03:37:05.953885071 (565h28m41.306670168s) [upload=497661]: Uploading segments... Sep 22 03:37:05.954033739 (565h28m41.306818837s) [upload=497661]: Got next segment. Inline: false Sep 22 03:37:05.954058936 (565h28m41.306844044s) [upload=497661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:37:05.954062213 (565h28m41.306847321s) [upload=497661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:37:05.978394584 (565h28m41.331179682s) [seg_pos={0 0}, upload=497661]: Begin upload segment... Sep 22 03:37:05.978424721 (565h28m41.331209829s) [seg_pos={0 0}, upload=497661]: Done begin upload segment. Sep 22 03:37:05.978443977 (565h28m41.331229085s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978484633 (565h28m41.331269741s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978762044 (565h28m41.331547153s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978783665 (565h28m41.331568773s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979021161 (565h28m41.331806269s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979040176 (565h28m41.331825274s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979239009 (565h28m41.332024117s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979346039 (565h28m41.332131147s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979977655 (565h28m41.332762763s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980195954 (565h28m41.332981053s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980470450 (565h28m41.333255558s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980850994 (565h28m41.333636092s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981390416 (565h28m41.334175524s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981549866 (565h28m41.334334974s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982114485 (565h28m41.334899593s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982508565 (565h28m41.335293663s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:06.088031568 (565h28m41.440816685s) [upload=497661]: Next returned nil segment Sep 22 03:37:06.088047888 (565h28m41.440832996s) [upload=497661]: Waiting for error group managing segments... Sep 22 03:37:06.097235449 (565h28m41.450020588s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098300347 (565h28m41.451085445s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098716739 (565h28m41.451501847s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098985253 (565h28m41.451770381s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099078658 (565h28m41.451863767s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099415269 (565h28m41.452200377s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099811354 (565h28m41.452596462s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099992072 (565h28m41.452777201s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100240468 (565h28m41.453025576s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100627725 (565h28m41.453412833s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100834132 (565h28m41.453619271s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.101022145 (565h28m41.453807254s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:35.942202163 (565h29m11.294987262s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942247639 (565h29m11.295032737s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942299857 (565h29m11.295084985s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942350993 (565h29m11.295136091s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942376942 (565h29m11.29516205s) [seg_pos={0 0}, upload=497661]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:37:35.957725684 (565h29m11.310510792s) [upload=497661]: Done uploading segments... Sep 22 03:37:35.957737848 (565h29m11.310522956s) [upload=497661]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 473.52719ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 476.236933ms FAIL (2025-09-22 03:38:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 476.259455ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....XX..........XX.....X..X...X..........X...X.XX...XXX...X.] duration: 404.97247ms FAIL (2025-09-22 03:38:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 405.077057ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.152958615s FAIL (2025-09-22 03:38:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.15298245s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....XX..........XX.....X..X...X..........X...X.XX...XXX...X] duration: 30.000799932s FAIL (2025-09-22 03:38:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00082506s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [......X.......XX............................................] duration: 2.419912929s FAIL (2025-09-22 03:37:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.419933207s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 542.054865ms FAIL (2025-09-22 03:37:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 542.077368ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X....................................X..X...X...X.X.] duration: 444.446174ms FAIL (2025-09-22 03:37:41 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:37:05.953774132 (565h28m41.30655924s) [upload=497661]: Starting upload Sep 22 03:37:05.953823264 (565h28m41.306608372s) [upload=497661]: Done starting upload Sep 22 03:37:05.953885071 (565h28m41.306670168s) [upload=497661]: Uploading segments... Sep 22 03:37:05.954033739 (565h28m41.306818837s) [upload=497661]: Got next segment. Inline: false Sep 22 03:37:05.954058936 (565h28m41.306844044s) [upload=497661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:37:05.954062213 (565h28m41.306847321s) [upload=497661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:37:05.978394584 (565h28m41.331179682s) [seg_pos={0 0}, upload=497661]: Begin upload segment... Sep 22 03:37:05.978424721 (565h28m41.331209829s) [seg_pos={0 0}, upload=497661]: Done begin upload segment. Sep 22 03:37:05.978443977 (565h28m41.331229085s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978484633 (565h28m41.331269741s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978762044 (565h28m41.331547153s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978783665 (565h28m41.331568773s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979021161 (565h28m41.331806269s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979040176 (565h28m41.331825274s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979239009 (565h28m41.332024117s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979346039 (565h28m41.332131147s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979977655 (565h28m41.332762763s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980195954 (565h28m41.332981053s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980470450 (565h28m41.333255558s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980850994 (565h28m41.333636092s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981390416 (565h28m41.334175524s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981549866 (565h28m41.334334974s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982114485 (565h28m41.334899593s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982508565 (565h28m41.335293663s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:06.088031568 (565h28m41.440816685s) [upload=497661]: Next returned nil segment Sep 22 03:37:06.088047888 (565h28m41.440832996s) [upload=497661]: Waiting for error group managing segments... Sep 22 03:37:06.097235449 (565h28m41.450020588s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098300347 (565h28m41.451085445s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098716739 (565h28m41.451501847s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098985253 (565h28m41.451770381s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099078658 (565h28m41.451863767s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099415269 (565h28m41.452200377s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099811354 (565h28m41.452596462s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099992072 (565h28m41.452777201s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100240468 (565h28m41.453025576s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100627725 (565h28m41.453412833s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100834132 (565h28m41.453619271s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.101022145 (565h28m41.453807254s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:35.942202163 (565h29m11.294987262s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942247639 (565h29m11.295032737s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942299857 (565h29m11.295084985s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942350993 (565h29m11.295136091s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942376942 (565h29m11.29516205s) [seg_pos={0 0}, upload=497661]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:37:35.957725684 (565h29m11.310510792s) [upload=497661]: Done uploading segments... Sep 22 03:37:35.957737848 (565h29m11.310522956s) [upload=497661]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.472213ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.499012483s FAIL (2025-09-22 03:37:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.499066314s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 730.585677ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:37:37 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 730.608038ms core satellite ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X........X....................................X..X...X...X.X] duration: 30.017656149s FAIL (2025-09-22 03:37:35 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:37:05.953774132 (565h28m41.30655924s) [upload=497661]: Starting upload Sep 22 03:37:05.953823264 (565h28m41.306608372s) [upload=497661]: Done starting upload Sep 22 03:37:05.953885071 (565h28m41.306670168s) [upload=497661]: Uploading segments... Sep 22 03:37:05.954033739 (565h28m41.306818837s) [upload=497661]: Got next segment. Inline: false Sep 22 03:37:05.954058936 (565h28m41.306844044s) [upload=497661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:37:05.954062213 (565h28m41.306847321s) [upload=497661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:37:05.978394584 (565h28m41.331179682s) [seg_pos={0 0}, upload=497661]: Begin upload segment... Sep 22 03:37:05.978424721 (565h28m41.331209829s) [seg_pos={0 0}, upload=497661]: Done begin upload segment. Sep 22 03:37:05.978443977 (565h28m41.331229085s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978484633 (565h28m41.331269741s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978762044 (565h28m41.331547153s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978783665 (565h28m41.331568773s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979021161 (565h28m41.331806269s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979040176 (565h28m41.331825274s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979239009 (565h28m41.332024117s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979346039 (565h28m41.332131147s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979977655 (565h28m41.332762763s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980195954 (565h28m41.332981053s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980470450 (565h28m41.333255558s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980850994 (565h28m41.333636092s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981390416 (565h28m41.334175524s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981549866 (565h28m41.334334974s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982114485 (565h28m41.334899593s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982508565 (565h28m41.335293663s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:06.088031568 (565h28m41.440816685s) [upload=497661]: Next returned nil segment Sep 22 03:37:06.088047888 (565h28m41.440832996s) [upload=497661]: Waiting for error group managing segments... Sep 22 03:37:06.097235449 (565h28m41.450020588s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098300347 (565h28m41.451085445s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098716739 (565h28m41.451501847s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098985253 (565h28m41.451770381s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099078658 (565h28m41.451863767s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099415269 (565h28m41.452200377s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099811354 (565h28m41.452596462s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099992072 (565h28m41.452777201s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100240468 (565h28m41.453025576s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100627725 (565h28m41.453412833s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100834132 (565h28m41.453619271s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.101022145 (565h28m41.453807254s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:35.942202163 (565h29m11.294987262s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942247639 (565h29m11.295032737s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942299857 (565h29m11.295084985s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942350993 (565h29m11.295136091s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942376942 (565h29m11.29516205s) [seg_pos={0 0}, upload=497661]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:37:35.957725684 (565h29m11.310510792s) [upload=497661]: Done uploading segments... Sep 22 03:37:35.957737848 (565h29m11.310522956s) [upload=497661]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.01767212s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......X.......XX...........................................] duration: 2.221394195s FAIL (2025-09-22 03:37:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.221414403s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 431.082228ms FAIL (2025-09-22 03:37:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 431.106242ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.026510726s FAIL (2025-09-22 03:36:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.026531695s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 742.501446ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:36:43 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 742.517097ms core satellite ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......XX..........XX.....X..X...X..........X...X.XX...XXX...] duration: 634.616284ms FAIL (2025-09-22 03:36:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 634.639158ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 612.188474ms FAIL (2025-09-22 03:36:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 612.214484ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [........X.......XX..........................................] duration: 2.20741278s FAIL (2025-09-22 03:36:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.207436764s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X........X....................................X..X...X...X.] duration: 388.322104ms FAIL (2025-09-22 03:36:12 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:35:30.210142434 (565h27m5.562927533s) [upload=497638]: Starting upload Sep 22 03:35:30.210204551 (565h27m5.562989649s) [upload=497638]: Done starting upload Sep 22 03:35:30.210297276 (565h27m5.563082384s) [upload=497638]: Uploading segments... Sep 22 03:35:30.210410748 (565h27m5.563195847s) [upload=497638]: Got next segment. Inline: false Sep 22 03:35:30.210454821 (565h27m5.56323992s) [upload=497638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:35:30.210458939 (565h27m5.563244037s) [upload=497638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:35:30.241333257 (565h27m5.594118355s) [seg_pos={0 0}, upload=497638]: Begin upload segment... Sep 22 03:35:30.241355479 (565h27m5.594140576s) [seg_pos={0 0}, upload=497638]: Done begin upload segment. Sep 22 03:35:30.241376698 (565h27m5.594161806s) [noise=true , address=45.91.116.13:4154 , piece_id=R2X53ULU7OTZV5XYAFJKMQZC4XJMESUBGBFMSZPILL6CQK5AKH5Q, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.241391356 (565h27m5.594176454s) [noise=true , address=145.79.227.18:4006 , piece_id=I2RNU4Q32UHVPZHTIFAV3AXT4SY6R6ALEKBPN4AZTDKU5AAGZCUA, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.241681791 (565h27m5.594466899s) [noise=true , address=77.90.31.16:4733 , piece_id=3KCJ46JODCMXMANRX5EWVR7JP6VDTL3564L4LQHZLE4DI34QZRIQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.241751922 (565h27m5.59453702s) [noise=true , address=45.91.116.7:4011 , piece_id=YUYLN5PPNIVYRDV5TQAB4CSWSEOTTSXNJCC5INMZRPKB6UULE3YQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.241921220 (565h27m5.594706317s) [noise=true , address=145.79.227.3:4151 , piece_id=XHWNWMMTSYUHCBSVEYXQEBWE7GJVPWNBGHYRI5FBJMWT4ZDMNJOQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242035585 (565h27m5.594820683s) [noise=true , address=45.91.116.10:4081 , piece_id=LDLL4CKBYHKW4UGBGB7WEPMMVZTXMU5XZU6GHJYCLYMKPFCXX5QA, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242150190 (565h27m5.594935288s) [noise=true , address=45.91.116.11:4118 , piece_id=DGAPEB6CPSCWE4V5NPDUXULABORG6SLZOEP2UTDQKFXR3VTVRKLQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242321872 (565h27m5.59510697s) [noise=true , address=45.91.116.11:4119 , piece_id=KYJLV3KOAECXZKNMPH4HSYGEWLEWQFQP7S3MO3U5EFHY5NQLKPNQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242385251 (565h27m5.595170359s) [noise=true , address=145.79.227.15:4875 , piece_id=QXOBPFTHW6ZQEWIA6CUASWR7JWRUMXMK7VPCGFPUBUQCQ5OJVRAQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242599803 (565h27m5.595384901s) [noise=true , address=145.79.227.18:4005 , piece_id=H4ZKGXEFUGBI2SPL3IZZWF6JKFDDST26SNM7WGRIDGOZ67Q3BSUA, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242623848 (565h27m5.595408956s) [noise=true , address=45.91.116.8:4045 , piece_id=3YS7NKPN2HK4ZADNMIS3QKXHLXBYI4XSEWRF3BHPQY65GUQBAQYA, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242904356 (565h27m5.595689454s) [noise=true , address=45.91.116.14:4191 , piece_id=GPH5WHZCZ5BUENLNWKVW7E2H6UISOAIW6H7WAYVJFSWOR7T6USGQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242916198 (565h27m5.595701306s) [noise=true , address=45.91.116.9:4263 , piece_id=IA63ROY5FRJS7SS6LBDZWH7CNQX4G4VRM4NB7GV4WLJPGQD6C2EQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.243253801 (565h27m5.596038909s) [noise=true , address=45.91.116.6:4975 , piece_id=PZWELPL2ITD7DA3DGE6ZNPUK6WV3LY3ARJS74VCGIQTZBM4X6ANQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.243284619 (565h27m5.596069717s) [noise=true , address=45.91.116.4:4937 , piece_id=3I7ZQNRKYAWMENMIIEWLKHOTJBLPDBJ4M7DKEVKBNWFVJOYK7QZA, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.244394191 (565h27m5.597179289s) [noise=true , address=77.90.31.12:4589 , piece_id=CCIK3LJ2ZDXHK7ZQ6WYS56OFAPR6MOMFQIZLXJIBKPHLSXVYVOUA, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.353215332 (565h27m5.70600046s) [upload=497638]: Next returned nil segment Sep 22 03:35:30.353230941 (565h27m5.706016069s) [upload=497638]: Waiting for error group managing segments... Sep 22 03:35:30.362428140 (565h27m5.715213279s) [noise=true , address=45.91.116.4:4937 , piece_id=3I7ZQNRKYAWMENMIIEWLKHOTJBLPDBJ4M7DKEVKBNWFVJOYK7QZA, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.363108878 (565h27m5.715894006s) [noise=true , address=45.91.116.11:4119 , piece_id=KYJLV3KOAECXZKNMPH4HSYGEWLEWQFQP7S3MO3U5EFHY5NQLKPNQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.363499992 (565h27m5.71628512s) [noise=true , address=45.91.116.9:4263 , piece_id=IA63ROY5FRJS7SS6LBDZWH7CNQX4G4VRM4NB7GV4WLJPGQD6C2EQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.363768145 (565h27m5.716553284s) [noise=true , address=45.91.116.13:4154 , piece_id=R2X53ULU7OTZV5XYAFJKMQZC4XJMESUBGBFMSZPILL6CQK5AKH5Q, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.364408637 (565h27m5.717193765s) [noise=true , address=45.91.116.10:4081 , piece_id=LDLL4CKBYHKW4UGBGB7WEPMMVZTXMU5XZU6GHJYCLYMKPFCXX5QA, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.364874632 (565h27m5.71765976s) [noise=true , address=45.91.116.14:4191 , piece_id=GPH5WHZCZ5BUENLNWKVW7E2H6UISOAIW6H7WAYVJFSWOR7T6USGQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.364990719 (565h27m5.717775848s) [noise=true , address=45.91.116.7:4011 , piece_id=YUYLN5PPNIVYRDV5TQAB4CSWSEOTTSXNJCC5INMZRPKB6UULE3YQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.365625121 (565h27m5.718410258s) [noise=true , address=45.91.116.8:4045 , piece_id=3YS7NKPN2HK4ZADNMIS3QKXHLXBYI4XSEWRF3BHPQY65GUQBAQYA, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.366107185 (565h27m5.718892322s) [noise=true , address=45.91.116.11:4118 , piece_id=DGAPEB6CPSCWE4V5NPDUXULABORG6SLZOEP2UTDQKFXR3VTVRKLQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.366197784 (565h27m5.718982913s) [noise=true , address=77.90.31.16:4733 , piece_id=3KCJ46JODCMXMANRX5EWVR7JP6VDTL3564L4LQHZLE4DI34QZRIQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.366762856 (565h27m5.719547983s) [noise=true , address=45.91.116.6:4975 , piece_id=PZWELPL2ITD7DA3DGE6ZNPUK6WV3LY3ARJS74VCGIQTZBM4X6ANQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.367255369 (565h27m5.720040497s) [noise=true , address=77.90.31.12:4589 , piece_id=CCIK3LJ2ZDXHK7ZQ6WYS56OFAPR6MOMFQIZLXJIBKPHLSXVYVOUA, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:36:00.195261453 (565h27m35.548046551s) [noise=true , address=145.79.227.18:4006 , piece_id=I2RNU4Q32UHVPZHTIFAV3AXT4SY6R6ALEKBPN4AZTDKU5AAGZCUA, seg_pos={0 0}, upload=497638]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: rpc: dial tcp 145.79.227.18:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:36:00.195292120 (565h27m35.548077218s) [noise=true , address=145.79.227.18:4005 , piece_id=H4ZKGXEFUGBI2SPL3IZZWF6JKFDDST26SNM7WGRIDGOZ67Q3BSUA, seg_pos={0 0}, upload=497638]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:36:00.195302560 (565h27m35.548087657s) [noise=true , address=145.79.227.3:4151 , piece_id=XHWNWMMTSYUHCBSVEYXQEBWE7GJVPWNBGHYRI5FBJMWT4ZDMNJOQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:36:00.195322799 (565h27m35.548107896s) [noise=true , address=145.79.227.15:4875 , piece_id=QXOBPFTHW6ZQEWIA6CUASWR7JWRUMXMK7VPCGFPUBUQCQ5OJVRAQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:36:00.195334490 (565h27m35.548119589s) [seg_pos={0 0}, upload=497638]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:36:00.212427877 (565h27m35.565212985s) [upload=497638]: Done uploading segments... Sep 22 03:36:00.212453666 (565h27m35.565238774s) [upload=497638]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 388.342572ms core uplink us-select-1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 827.366306ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:36:08 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 827.385012ms core satellite ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X........X....................................X..X...X...X] duration: 30.01837195s FAIL (2025-09-22 03:36:00 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:35:30.210142434 (565h27m5.562927533s) [upload=497638]: Starting upload Sep 22 03:35:30.210204551 (565h27m5.562989649s) [upload=497638]: Done starting upload Sep 22 03:35:30.210297276 (565h27m5.563082384s) [upload=497638]: Uploading segments... Sep 22 03:35:30.210410748 (565h27m5.563195847s) [upload=497638]: Got next segment. Inline: false Sep 22 03:35:30.210454821 (565h27m5.56323992s) [upload=497638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:35:30.210458939 (565h27m5.563244037s) [upload=497638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:35:30.241333257 (565h27m5.594118355s) [seg_pos={0 0}, upload=497638]: Begin upload segment... Sep 22 03:35:30.241355479 (565h27m5.594140576s) [seg_pos={0 0}, upload=497638]: Done begin upload segment. Sep 22 03:35:30.241376698 (565h27m5.594161806s) [noise=true , address=45.91.116.13:4154 , piece_id=R2X53ULU7OTZV5XYAFJKMQZC4XJMESUBGBFMSZPILL6CQK5AKH5Q, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.241391356 (565h27m5.594176454s) [noise=true , address=145.79.227.18:4006 , piece_id=I2RNU4Q32UHVPZHTIFAV3AXT4SY6R6ALEKBPN4AZTDKU5AAGZCUA, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.241681791 (565h27m5.594466899s) [noise=true , address=77.90.31.16:4733 , piece_id=3KCJ46JODCMXMANRX5EWVR7JP6VDTL3564L4LQHZLE4DI34QZRIQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.241751922 (565h27m5.59453702s) [noise=true , address=45.91.116.7:4011 , piece_id=YUYLN5PPNIVYRDV5TQAB4CSWSEOTTSXNJCC5INMZRPKB6UULE3YQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.241921220 (565h27m5.594706317s) [noise=true , address=145.79.227.3:4151 , piece_id=XHWNWMMTSYUHCBSVEYXQEBWE7GJVPWNBGHYRI5FBJMWT4ZDMNJOQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242035585 (565h27m5.594820683s) [noise=true , address=45.91.116.10:4081 , piece_id=LDLL4CKBYHKW4UGBGB7WEPMMVZTXMU5XZU6GHJYCLYMKPFCXX5QA, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242150190 (565h27m5.594935288s) [noise=true , address=45.91.116.11:4118 , piece_id=DGAPEB6CPSCWE4V5NPDUXULABORG6SLZOEP2UTDQKFXR3VTVRKLQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242321872 (565h27m5.59510697s) [noise=true , address=45.91.116.11:4119 , piece_id=KYJLV3KOAECXZKNMPH4HSYGEWLEWQFQP7S3MO3U5EFHY5NQLKPNQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242385251 (565h27m5.595170359s) [noise=true , address=145.79.227.15:4875 , piece_id=QXOBPFTHW6ZQEWIA6CUASWR7JWRUMXMK7VPCGFPUBUQCQ5OJVRAQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242599803 (565h27m5.595384901s) [noise=true , address=145.79.227.18:4005 , piece_id=H4ZKGXEFUGBI2SPL3IZZWF6JKFDDST26SNM7WGRIDGOZ67Q3BSUA, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242623848 (565h27m5.595408956s) [noise=true , address=45.91.116.8:4045 , piece_id=3YS7NKPN2HK4ZADNMIS3QKXHLXBYI4XSEWRF3BHPQY65GUQBAQYA, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242904356 (565h27m5.595689454s) [noise=true , address=45.91.116.14:4191 , piece_id=GPH5WHZCZ5BUENLNWKVW7E2H6UISOAIW6H7WAYVJFSWOR7T6USGQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.242916198 (565h27m5.595701306s) [noise=true , address=45.91.116.9:4263 , piece_id=IA63ROY5FRJS7SS6LBDZWH7CNQX4G4VRM4NB7GV4WLJPGQD6C2EQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.243253801 (565h27m5.596038909s) [noise=true , address=45.91.116.6:4975 , piece_id=PZWELPL2ITD7DA3DGE6ZNPUK6WV3LY3ARJS74VCGIQTZBM4X6ANQ, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.243284619 (565h27m5.596069717s) [noise=true , address=45.91.116.4:4937 , piece_id=3I7ZQNRKYAWMENMIIEWLKHOTJBLPDBJ4M7DKEVKBNWFVJOYK7QZA, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.244394191 (565h27m5.597179289s) [noise=true , address=77.90.31.12:4589 , piece_id=CCIK3LJ2ZDXHK7ZQ6WYS56OFAPR6MOMFQIZLXJIBKPHLSXVYVOUA, seg_pos={0 0}, upload=497638]: Uploading piece... Sep 22 03:35:30.353215332 (565h27m5.70600046s) [upload=497638]: Next returned nil segment Sep 22 03:35:30.353230941 (565h27m5.706016069s) [upload=497638]: Waiting for error group managing segments... Sep 22 03:35:30.362428140 (565h27m5.715213279s) [noise=true , address=45.91.116.4:4937 , piece_id=3I7ZQNRKYAWMENMIIEWLKHOTJBLPDBJ4M7DKEVKBNWFVJOYK7QZA, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.363108878 (565h27m5.715894006s) [noise=true , address=45.91.116.11:4119 , piece_id=KYJLV3KOAECXZKNMPH4HSYGEWLEWQFQP7S3MO3U5EFHY5NQLKPNQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.363499992 (565h27m5.71628512s) [noise=true , address=45.91.116.9:4263 , piece_id=IA63ROY5FRJS7SS6LBDZWH7CNQX4G4VRM4NB7GV4WLJPGQD6C2EQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.363768145 (565h27m5.716553284s) [noise=true , address=45.91.116.13:4154 , piece_id=R2X53ULU7OTZV5XYAFJKMQZC4XJMESUBGBFMSZPILL6CQK5AKH5Q, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.364408637 (565h27m5.717193765s) [noise=true , address=45.91.116.10:4081 , piece_id=LDLL4CKBYHKW4UGBGB7WEPMMVZTXMU5XZU6GHJYCLYMKPFCXX5QA, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.364874632 (565h27m5.71765976s) [noise=true , address=45.91.116.14:4191 , piece_id=GPH5WHZCZ5BUENLNWKVW7E2H6UISOAIW6H7WAYVJFSWOR7T6USGQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.364990719 (565h27m5.717775848s) [noise=true , address=45.91.116.7:4011 , piece_id=YUYLN5PPNIVYRDV5TQAB4CSWSEOTTSXNJCC5INMZRPKB6UULE3YQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.365625121 (565h27m5.718410258s) [noise=true , address=45.91.116.8:4045 , piece_id=3YS7NKPN2HK4ZADNMIS3QKXHLXBYI4XSEWRF3BHPQY65GUQBAQYA, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.366107185 (565h27m5.718892322s) [noise=true , address=45.91.116.11:4118 , piece_id=DGAPEB6CPSCWE4V5NPDUXULABORG6SLZOEP2UTDQKFXR3VTVRKLQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.366197784 (565h27m5.718982913s) [noise=true , address=77.90.31.16:4733 , piece_id=3KCJ46JODCMXMANRX5EWVR7JP6VDTL3564L4LQHZLE4DI34QZRIQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.366762856 (565h27m5.719547983s) [noise=true , address=45.91.116.6:4975 , piece_id=PZWELPL2ITD7DA3DGE6ZNPUK6WV3LY3ARJS74VCGIQTZBM4X6ANQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:35:30.367255369 (565h27m5.720040497s) [noise=true , address=77.90.31.12:4589 , piece_id=CCIK3LJ2ZDXHK7ZQ6WYS56OFAPR6MOMFQIZLXJIBKPHLSXVYVOUA, seg_pos={0 0}, upload=497638]: Done uploading piece. err: <nil> Sep 22 03:36:00.195261453 (565h27m35.548046551s) [noise=true , address=145.79.227.18:4006 , piece_id=I2RNU4Q32UHVPZHTIFAV3AXT4SY6R6ALEKBPN4AZTDKU5AAGZCUA, seg_pos={0 0}, upload=497638]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: rpc: dial tcp 145.79.227.18:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:36:00.195292120 (565h27m35.548077218s) [noise=true , address=145.79.227.18:4005 , piece_id=H4ZKGXEFUGBI2SPL3IZZWF6JKFDDST26SNM7WGRIDGOZ67Q3BSUA, seg_pos={0 0}, upload=497638]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:36:00.195302560 (565h27m35.548087657s) [noise=true , address=145.79.227.3:4151 , piece_id=XHWNWMMTSYUHCBSVEYXQEBWE7GJVPWNBGHYRI5FBJMWT4ZDMNJOQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:36:00.195322799 (565h27m35.548107896s) [noise=true , address=145.79.227.15:4875 , piece_id=QXOBPFTHW6ZQEWIA6CUASWR7JWRUMXMK7VPCGFPUBUQCQ5OJVRAQ, seg_pos={0 0}, upload=497638]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:36:00.195334490 (565h27m35.548119589s) [seg_pos={0 0}, upload=497638]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:36:00.212427877 (565h27m35.565212985s) [upload=497638]: Done uploading segments... Sep 22 03:36:00.212453666 (565h27m35.565238774s) [upload=497638]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.01839327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 412.824835ms FAIL (2025-09-22 03:35:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 412.849091ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 748.458857ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:35:34 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 748.479826ms core satellite ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.093018722s FAIL (2025-09-22 03:35:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.093051434s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......XX..........XX.....X..X...X..........X...X.XX...XXX..] duration: 430.750114ms FAIL (2025-09-22 03:35:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 430.773068ms edge gateway us-select-1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 763.774687ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:35:05 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 763.794635ms core satellite ap1

Orbiter Check History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)