Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......XX.....X..X...X..........X...X.XX...XXX...X.......X..] duration: 399.193185ms FAIL (2025-09-22 03:45:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 399.406766ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................................X..X...X...X.X.....XX.....X] duration: 30.021685245s FAIL (2025-09-22 03:45:44 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:45:14.747159999 (565h36m50.099945137s) [upload=497780]: Starting upload Sep 22 03:45:14.747238176 (565h36m50.100023304s) [upload=497780]: Done starting upload Sep 22 03:45:14.747303748 (565h36m50.100088887s) [upload=497780]: Uploading segments... Sep 22 03:45:14.747706776 (565h36m50.100491904s) [upload=497780]: Got next segment. Inline: false Sep 22 03:45:14.747755707 (565h36m50.100540836s) [upload=497780]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:45:14.747761568 (565h36m50.100546697s) [upload=497780]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:45:14.771470018 (565h36m50.124255147s) [seg_pos={0 0}, upload=497780]: Begin upload segment... Sep 22 03:45:14.771520885 (565h36m50.124306022s) [seg_pos={0 0}, upload=497780]: Done begin upload segment. Sep 22 03:45:14.771552664 (565h36m50.124337792s) [noise=true , address=77.90.31.14:4661 , piece_id=GENOSX3OC7EUOJVVKD3UUNTIKEZQAI5LNZ4DJBEMAKFQRODIIO4A, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.771561370 (565h36m50.124346498s) [noise=true , address=45.91.116.10:4083 , piece_id=NCB3MNPDEOJCM5DKQMKS5OZA73OKFTZ34W3DQV23JRUASN3T5EUA, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.772081988 (565h36m50.124867115s) [noise=true , address=45.91.116.11:4119 , piece_id=NLBNS35XXU7J7JMIRABNUJOVTJ27UY5RY2D76TA6AIX3THFYVBQA, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.772112735 (565h36m50.124897864s) [noise=true , address=77.90.31.21:4877 , piece_id=CWBSNKKH2AA7LNONDMJDA34WRLELYZW5JD4MUMK34FTZHPGACCNA, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.772494661 (565h36m50.1252798s) [noise=true , address=45.91.116.14:4189 , piece_id=LLAJ7XCR4YI5EMZ6PZUIGXEOR3RMTELVHDB76Q2W6EPFRDLFYIUQ, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.772633292 (565h36m50.12541842s) [noise=true , address=145.79.227.6:4258 , piece_id=VYCY4JXKWTZIALRQKXYGO4ZRKYH2KVNNET6RXDSD2YO7XA3C2CKA, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.772999660 (565h36m50.125784788s) [noise=true , address=77.90.31.16:4735 , piece_id=J2WQCOSEDL74AGFJM2Y672PTYYOCFLJLSFSEQAPTRIYW4QQTF4CA, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.773159429 (565h36m50.125944567s) [noise=true , address=77.90.31.17:4770 , piece_id=CGEGRY33B4K7QQYH77OWW3KCDBM5DASUL7S25HDF5J5UJ7FKSLIA, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.773437190 (565h36m50.126222318s) [noise=true , address=77.90.31.19:4843 , piece_id=QODTZDEKUM2EY4LWWQCBP7OPQNX3NW6NEXUWKCM5BIXNEVBWJ5LQ, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.773533090 (565h36m50.126318218s) [noise=true , address=145.79.227.16:4909 , piece_id=EBCDPN6E6AJOHVADLXSBJXD5EJLDJGEVM7MKIQGXN5F35CDKKAAA, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.773873380 (565h36m50.126658507s) [noise=true , address=45.91.116.8:4047 , piece_id=UBJK4HXMSPB4OMYPGSB7GRDQOLBT7KLMKGQV2BZU5ZBTU3LVGDKA, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.773912132 (565h36m50.12669726s) [noise=true , address=45.91.116.8:4045 , piece_id=YYOAOI5ZJJU2IZGVZJRJ7MAK5CMSA5ILHGG54VO2NUPBCHDCMTPQ, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.774259313 (565h36m50.127044452s) [noise=true , address=77.90.31.14:4662 , piece_id=IGEEK23TC45MDMUQQUQK3GE6LHB73DQMD4O754V42WAGN3KIFGVQ, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.774278519 (565h36m50.127063648s) [noise=true , address=45.91.116.14:4190 , piece_id=33A6GFDAKSKGHKRDDL2RWPDO263ZKMNGALOHTRSV4BZIHR3UR3OA, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.774604381 (565h36m50.12738951s) [noise=true , address=145.79.227.17:4969 , piece_id=DLLFH7V5GFJKNFBPG6OMONLINRZ75HMRA6LD777T3J2R6RF24MBA, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.774974786 (565h36m50.127759914s) [noise=true , address=145.79.227.11:4730 , piece_id=5DP7JZLLVMVSK52OBTQCM7GEO7TMDWBYYCKOWRTOIKXPLRHLACMA, seg_pos={0 0}, upload=497780]: Uploading piece... Sep 22 03:45:14.892566525 (565h36m50.245351633s) [upload=497780]: Next returned nil segment Sep 22 03:45:14.892580211 (565h36m50.245365319s) [upload=497780]: Waiting for error group managing segments... Sep 22 03:45:14.901916712 (565h36m50.25470182s) [noise=true , address=45.91.116.8:4045 , piece_id=YYOAOI5ZJJU2IZGVZJRJ7MAK5CMSA5ILHGG54VO2NUPBCHDCMTPQ, seg_pos={0 0}, upload=497780]: Done uploading piece. err: <nil> Sep 22 03:45:14.902525644 (565h36m50.255310752s) [noise=true , address=45.91.116.14:4189 , piece_id=LLAJ7XCR4YI5EMZ6PZUIGXEOR3RMTELVHDB76Q2W6EPFRDLFYIUQ, seg_pos={0 0}, upload=497780]: Done uploading piece. err: <nil> Sep 22 03:45:14.902985557 (565h36m50.255770675s) [noise=true , address=45.91.116.8:4047 , piece_id=UBJK4HXMSPB4OMYPGSB7GRDQOLBT7KLMKGQV2BZU5ZBTU3LVGDKA, seg_pos={0 0}, upload=497780]: Done uploading piece. err: <nil> Sep 22 03:45:14.903310025 (565h36m50.256095154s) [noise=true , address=45.91.116.14:4190 , piece_id=33A6GFDAKSKGHKRDDL2RWPDO263ZKMNGALOHTRSV4BZIHR3UR3OA, seg_pos={0 0}, upload=497780]: Done uploading piece. err: <nil> Sep 22 03:45:14.903452994 (565h36m50.256238102s) [noise=true , address=45.91.116.10:4083 , piece_id=NCB3MNPDEOJCM5DKQMKS5OZA73OKFTZ34W3DQV23JRUASN3T5EUA, seg_pos={0 0}, upload=497780]: Done uploading piece. err: <nil> Sep 22 03:45:14.904038263 (565h36m50.256823391s) [noise=true , address=45.91.116.11:4119 , piece_id=NLBNS35XXU7J7JMIRABNUJOVTJ27UY5RY2D76TA6AIX3THFYVBQA, seg_pos={0 0}, upload=497780]: Done uploading piece. err: <nil> Sep 22 03:45:14.904499398 (565h36m50.257284506s) [noise=true , address=77.90.31.19:4843 , piece_id=QODTZDEKUM2EY4LWWQCBP7OPQNX3NW6NEXUWKCM5BIXNEVBWJ5LQ, seg_pos={0 0}, upload=497780]: Done uploading piece. err: <nil> Sep 22 03:45:14.904711646 (565h36m50.257496764s) [noise=true , address=77.90.31.14:4662 , piece_id=IGEEK23TC45MDMUQQUQK3GE6LHB73DQMD4O754V42WAGN3KIFGVQ, seg_pos={0 0}, upload=497780]: Done uploading piece. err: <nil> Sep 22 03:45:14.905216584 (565h36m50.258001702s) [noise=true , address=77.90.31.14:4661 , piece_id=GENOSX3OC7EUOJVVKD3UUNTIKEZQAI5LNZ4DJBEMAKFQRODIIO4A, seg_pos={0 0}, upload=497780]: Done uploading piece. err: <nil> Sep 22 03:45:14.905229238 (565h36m50.258014365s) [noise=true , address=77.90.31.16:4735 , piece_id=J2WQCOSEDL74AGFJM2Y672PTYYOCFLJLSFSEQAPTRIYW4QQTF4CA, seg_pos={0 0}, upload=497780]: Done uploading piece. err: <nil> Sep 22 03:45:14.905787776 (565h36m50.258572904s) [noise=true , address=77.90.31.21:4877 , piece_id=CWBSNKKH2AA7LNONDMJDA34WRLELYZW5JD4MUMK34FTZHPGACCNA, seg_pos={0 0}, upload=497780]: Done uploading piece. err: <nil> Sep 22 03:45:14.905895137 (565h36m50.258680245s) [noise=true , address=77.90.31.17:4770 , piece_id=CGEGRY33B4K7QQYH77OWW3KCDBM5DASUL7S25HDF5J5UJ7FKSLIA, seg_pos={0 0}, upload=497780]: Done uploading piece. err: <nil> Sep 22 03:45:31.204265371 (565h37m6.55705051s) [noise=true , address=145.79.227.16:4909 , piece_id=EBCDPN6E6AJOHVADLXSBJXD5EJLDJGEVM7MKIQGXN5F35CDKKAAA, seg_pos={0 0}, upload=497780]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op Sep 22 03:45:44.733652533 (565h37m20.086437641s) [noise=true , address=145.79.227.11:4730 , piece_id=5DP7JZLLVMVSK52OBTQCM7GEO7TMDWBYYCKOWRTOIKXPLRHLACMA, seg_pos={0 0}, upload=497780]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op Sep 22 03:45:44.733668112 (565h37m20.08645322s) [noise=true , address=145.79.227.17:4969 , piece_id=DLLFH7V5GFJKNFBPG6OMONLINRZ75HMRA6LD777T3J2R6RF24MBA, seg_pos={0 0}, upload=497780]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:45:44.733780202 (565h37m20.0865653s) [noise=true , address=145.79.227.6:4258 , piece_id=VYCY4JXKWTZIALRQKXYGO4ZRKYH2KVNNET6RXDSD2YO7XA3C2CKA, seg_pos={0 0}, upload=497780]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:45:44.733802023 (565h37m20.086587131s) [seg_pos={0 0}, upload=497780]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:45:44.753762991 (565h37m20.106548089s) [upload=497780]: Done uploading segments... Sep 22 03:45:44.753775224 (565h37m20.106560332s) [upload=497780]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.021702778s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 2.120256062s FAIL (2025-09-22 03:45:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.120287371s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 2.128643172s FAIL (2025-09-22 03:45:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.12867974s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........XX.....X..X...X..........X...X.XX...XXX...X.......X.] duration: 421.755743ms FAIL (2025-09-22 03:44:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 421.773748ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................................X..X...X...X.X.....XX.....] duration: 426.753638ms FAIL (2025-09-22 03:44:37 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:41:53.821360598 (565h33m29.174145726s) [upload=497735]: Starting upload Sep 22 03:41:53.821432342 (565h33m29.174217471s) [upload=497735]: Done starting upload Sep 22 03:41:53.821573697 (565h33m29.174358826s) [upload=497735]: Uploading segments... Sep 22 03:41:53.821895693 (565h33m29.174680821s) [upload=497735]: Got next segment. Inline: false Sep 22 03:41:53.821989369 (565h33m29.174774496s) [upload=497735]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:41:53.821995941 (565h33m29.174781069s) [upload=497735]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:41:53.843161018 (565h33m29.195946156s) [seg_pos={0 0}, upload=497735]: Begin upload segment... Sep 22 03:41:53.843219267 (565h33m29.196004405s) [seg_pos={0 0}, upload=497735]: Done begin upload segment. Sep 22 03:41:53.843260845 (565h33m29.196045973s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843270974 (565h33m29.196056102s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843790710 (565h33m29.196575838s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844019850 (565h33m29.196804988s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844360168 (565h33m29.197145296s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844472439 (565h33m29.197257576s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844820201 (565h33m29.19760533s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844922043 (565h33m29.197707171s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845271628 (565h33m29.198056767s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845900639 (565h33m29.198685767s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846045771 (565h33m29.1988309s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846725577 (565h33m29.199510705s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847449616 (565h33m29.200234744s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847961646 (565h33m29.200746775s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.849104080 (565h33m29.201889208s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.850005882 (565h33m29.202791011s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.982635031 (565h33m29.335420168s) [upload=497735]: Next returned nil segment Sep 22 03:41:53.982652824 (565h33m29.335437952s) [upload=497735]: Waiting for error group managing segments... Sep 22 03:41:53.990044095 (565h33m29.342829232s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.990531790 (565h33m29.343316918s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991047046 (565h33m29.343832184s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991466263 (565h33m29.344251392s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991769693 (565h33m29.344554831s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992458005 (565h33m29.345243133s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992505003 (565h33m29.345290131s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993159922 (565h33m29.34594505s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993274718 (565h33m29.346059845s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994692838 (565h33m29.347477966s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994841236 (565h33m29.347626365s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.995361182 (565h33m29.348146311s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:42:23.809279587 (565h33m59.162064714s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809421593 (565h33m59.162206721s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809560264 (565h33m59.162345402s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:42:23.809598205 (565h33m59.162383333s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; context deadline exceeded Sep 22 03:42:23.809711137 (565h33m59.162496265s) [seg_pos={0 0}, upload=497735]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:42:23.825581299 (565h33m59.178366426s) [upload=497735]: Done uploading segments... Sep 22 03:42:23.825600274 (565h33m59.178385412s) [upload=497735]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 426.78075ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 4.926941378s FAIL (2025-09-22 03:44:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.926961085s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 2.426183507s FAIL (2025-09-22 03:44:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.426222601s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................................X..X...X...X.X.....XX....] duration: 496.225633ms FAIL (2025-09-22 03:44:12 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:41:53.821360598 (565h33m29.174145726s) [upload=497735]: Starting upload Sep 22 03:41:53.821432342 (565h33m29.174217471s) [upload=497735]: Done starting upload Sep 22 03:41:53.821573697 (565h33m29.174358826s) [upload=497735]: Uploading segments... Sep 22 03:41:53.821895693 (565h33m29.174680821s) [upload=497735]: Got next segment. Inline: false Sep 22 03:41:53.821989369 (565h33m29.174774496s) [upload=497735]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:41:53.821995941 (565h33m29.174781069s) [upload=497735]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:41:53.843161018 (565h33m29.195946156s) [seg_pos={0 0}, upload=497735]: Begin upload segment... Sep 22 03:41:53.843219267 (565h33m29.196004405s) [seg_pos={0 0}, upload=497735]: Done begin upload segment. Sep 22 03:41:53.843260845 (565h33m29.196045973s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843270974 (565h33m29.196056102s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843790710 (565h33m29.196575838s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844019850 (565h33m29.196804988s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844360168 (565h33m29.197145296s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844472439 (565h33m29.197257576s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844820201 (565h33m29.19760533s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844922043 (565h33m29.197707171s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845271628 (565h33m29.198056767s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845900639 (565h33m29.198685767s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846045771 (565h33m29.1988309s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846725577 (565h33m29.199510705s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847449616 (565h33m29.200234744s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847961646 (565h33m29.200746775s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.849104080 (565h33m29.201889208s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.850005882 (565h33m29.202791011s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.982635031 (565h33m29.335420168s) [upload=497735]: Next returned nil segment Sep 22 03:41:53.982652824 (565h33m29.335437952s) [upload=497735]: Waiting for error group managing segments... Sep 22 03:41:53.990044095 (565h33m29.342829232s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.990531790 (565h33m29.343316918s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991047046 (565h33m29.343832184s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991466263 (565h33m29.344251392s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991769693 (565h33m29.344554831s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992458005 (565h33m29.345243133s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992505003 (565h33m29.345290131s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993159922 (565h33m29.34594505s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993274718 (565h33m29.346059845s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994692838 (565h33m29.347477966s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994841236 (565h33m29.347626365s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.995361182 (565h33m29.348146311s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:42:23.809279587 (565h33m59.162064714s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809421593 (565h33m59.162206721s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809560264 (565h33m59.162345402s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:42:23.809598205 (565h33m59.162383333s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; context deadline exceeded Sep 22 03:42:23.809711137 (565h33m59.162496265s) [seg_pos={0 0}, upload=497735]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:42:23.825581299 (565h33m59.178366426s) [upload=497735]: Done uploading segments... Sep 22 03:42:23.825600274 (565h33m59.178385412s) [upload=497735]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 496.255279ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.........XX.....X..X...X..........X...X.XX...XXX...X.......X] duration: 30.000755072s FAIL (2025-09-22 03:44:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000767375s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................................X..X...X...X.X.....XX...] duration: 447.612737ms FAIL (2025-09-22 03:43:52 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:41:53.821360598 (565h33m29.174145726s) [upload=497735]: Starting upload Sep 22 03:41:53.821432342 (565h33m29.174217471s) [upload=497735]: Done starting upload Sep 22 03:41:53.821573697 (565h33m29.174358826s) [upload=497735]: Uploading segments... Sep 22 03:41:53.821895693 (565h33m29.174680821s) [upload=497735]: Got next segment. Inline: false Sep 22 03:41:53.821989369 (565h33m29.174774496s) [upload=497735]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:41:53.821995941 (565h33m29.174781069s) [upload=497735]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:41:53.843161018 (565h33m29.195946156s) [seg_pos={0 0}, upload=497735]: Begin upload segment... Sep 22 03:41:53.843219267 (565h33m29.196004405s) [seg_pos={0 0}, upload=497735]: Done begin upload segment. Sep 22 03:41:53.843260845 (565h33m29.196045973s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843270974 (565h33m29.196056102s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843790710 (565h33m29.196575838s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844019850 (565h33m29.196804988s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844360168 (565h33m29.197145296s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844472439 (565h33m29.197257576s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844820201 (565h33m29.19760533s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844922043 (565h33m29.197707171s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845271628 (565h33m29.198056767s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845900639 (565h33m29.198685767s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846045771 (565h33m29.1988309s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846725577 (565h33m29.199510705s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847449616 (565h33m29.200234744s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847961646 (565h33m29.200746775s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.849104080 (565h33m29.201889208s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.850005882 (565h33m29.202791011s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.982635031 (565h33m29.335420168s) [upload=497735]: Next returned nil segment Sep 22 03:41:53.982652824 (565h33m29.335437952s) [upload=497735]: Waiting for error group managing segments... Sep 22 03:41:53.990044095 (565h33m29.342829232s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.990531790 (565h33m29.343316918s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991047046 (565h33m29.343832184s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991466263 (565h33m29.344251392s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991769693 (565h33m29.344554831s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992458005 (565h33m29.345243133s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992505003 (565h33m29.345290131s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993159922 (565h33m29.34594505s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993274718 (565h33m29.346059845s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994692838 (565h33m29.347477966s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994841236 (565h33m29.347626365s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.995361182 (565h33m29.348146311s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:42:23.809279587 (565h33m59.162064714s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809421593 (565h33m59.162206721s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809560264 (565h33m59.162345402s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:42:23.809598205 (565h33m59.162383333s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; context deadline exceeded Sep 22 03:42:23.809711137 (565h33m59.162496265s) [seg_pos={0 0}, upload=497735]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:42:23.825581299 (565h33m59.178366426s) [upload=497735]: Done uploading segments... Sep 22 03:42:23.825600274 (565h33m59.178385412s) [upload=497735]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 447.635571ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 2.189527545s FAIL (2025-09-22 03:43:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.189552482s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........XX.....X..X...X..........X...X.XX...XXX...X.......] duration: 604.504728ms FAIL (2025-09-22 03:43:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 604.522241ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X....................................X..X...X...X.X.....XX..] duration: 494.818802ms FAIL (2025-09-22 03:43:10 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:41:53.821360598 (565h33m29.174145726s) [upload=497735]: Starting upload Sep 22 03:41:53.821432342 (565h33m29.174217471s) [upload=497735]: Done starting upload Sep 22 03:41:53.821573697 (565h33m29.174358826s) [upload=497735]: Uploading segments... Sep 22 03:41:53.821895693 (565h33m29.174680821s) [upload=497735]: Got next segment. Inline: false Sep 22 03:41:53.821989369 (565h33m29.174774496s) [upload=497735]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:41:53.821995941 (565h33m29.174781069s) [upload=497735]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:41:53.843161018 (565h33m29.195946156s) [seg_pos={0 0}, upload=497735]: Begin upload segment... Sep 22 03:41:53.843219267 (565h33m29.196004405s) [seg_pos={0 0}, upload=497735]: Done begin upload segment. Sep 22 03:41:53.843260845 (565h33m29.196045973s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843270974 (565h33m29.196056102s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843790710 (565h33m29.196575838s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844019850 (565h33m29.196804988s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844360168 (565h33m29.197145296s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844472439 (565h33m29.197257576s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844820201 (565h33m29.19760533s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844922043 (565h33m29.197707171s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845271628 (565h33m29.198056767s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845900639 (565h33m29.198685767s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846045771 (565h33m29.1988309s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846725577 (565h33m29.199510705s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847449616 (565h33m29.200234744s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847961646 (565h33m29.200746775s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.849104080 (565h33m29.201889208s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.850005882 (565h33m29.202791011s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.982635031 (565h33m29.335420168s) [upload=497735]: Next returned nil segment Sep 22 03:41:53.982652824 (565h33m29.335437952s) [upload=497735]: Waiting for error group managing segments... Sep 22 03:41:53.990044095 (565h33m29.342829232s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.990531790 (565h33m29.343316918s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991047046 (565h33m29.343832184s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991466263 (565h33m29.344251392s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991769693 (565h33m29.344554831s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992458005 (565h33m29.345243133s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992505003 (565h33m29.345290131s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993159922 (565h33m29.34594505s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993274718 (565h33m29.346059845s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994692838 (565h33m29.347477966s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994841236 (565h33m29.347626365s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.995361182 (565h33m29.348146311s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:42:23.809279587 (565h33m59.162064714s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809421593 (565h33m59.162206721s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809560264 (565h33m59.162345402s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:42:23.809598205 (565h33m59.162383333s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; context deadline exceeded Sep 22 03:42:23.809711137 (565h33m59.162496265s) [seg_pos={0 0}, upload=497735]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:42:23.825581299 (565h33m59.178366426s) [upload=497735]: Done uploading segments... Sep 22 03:42:23.825600274 (565h33m59.178385412s) [upload=497735]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 494.841886ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.063189407s FAIL (2025-09-22 03:42:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.06321263s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [X.......XX..................................................] duration: 2.240454704s FAIL (2025-09-22 03:42:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.240479621s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X....................................X..X...X...X.X.....XX.] duration: 715.509473ms FAIL (2025-09-22 03:42:24 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:41:53.821360598 (565h33m29.174145726s) [upload=497735]: Starting upload Sep 22 03:41:53.821432342 (565h33m29.174217471s) [upload=497735]: Done starting upload Sep 22 03:41:53.821573697 (565h33m29.174358826s) [upload=497735]: Uploading segments... Sep 22 03:41:53.821895693 (565h33m29.174680821s) [upload=497735]: Got next segment. Inline: false Sep 22 03:41:53.821989369 (565h33m29.174774496s) [upload=497735]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:41:53.821995941 (565h33m29.174781069s) [upload=497735]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:41:53.843161018 (565h33m29.195946156s) [seg_pos={0 0}, upload=497735]: Begin upload segment... Sep 22 03:41:53.843219267 (565h33m29.196004405s) [seg_pos={0 0}, upload=497735]: Done begin upload segment. Sep 22 03:41:53.843260845 (565h33m29.196045973s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843270974 (565h33m29.196056102s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843790710 (565h33m29.196575838s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844019850 (565h33m29.196804988s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844360168 (565h33m29.197145296s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844472439 (565h33m29.197257576s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844820201 (565h33m29.19760533s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844922043 (565h33m29.197707171s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845271628 (565h33m29.198056767s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845900639 (565h33m29.198685767s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846045771 (565h33m29.1988309s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846725577 (565h33m29.199510705s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847449616 (565h33m29.200234744s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847961646 (565h33m29.200746775s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.849104080 (565h33m29.201889208s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.850005882 (565h33m29.202791011s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.982635031 (565h33m29.335420168s) [upload=497735]: Next returned nil segment Sep 22 03:41:53.982652824 (565h33m29.335437952s) [upload=497735]: Waiting for error group managing segments... Sep 22 03:41:53.990044095 (565h33m29.342829232s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.990531790 (565h33m29.343316918s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991047046 (565h33m29.343832184s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991466263 (565h33m29.344251392s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991769693 (565h33m29.344554831s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992458005 (565h33m29.345243133s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992505003 (565h33m29.345290131s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993159922 (565h33m29.34594505s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993274718 (565h33m29.346059845s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994692838 (565h33m29.347477966s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994841236 (565h33m29.347626365s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.995361182 (565h33m29.348146311s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:42:23.809279587 (565h33m59.162064714s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809421593 (565h33m59.162206721s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809560264 (565h33m59.162345402s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:42:23.809598205 (565h33m59.162383333s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; context deadline exceeded Sep 22 03:42:23.809711137 (565h33m59.162496265s) [seg_pos={0 0}, upload=497735]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:42:23.825581299 (565h33m59.178366426s) [upload=497735]: Done uploading segments... Sep 22 03:42:23.825600274 (565h33m59.178385412s) [upload=497735]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 715.541174ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X....................................X..X...X...X.X.....XX] duration: 30.017879052s FAIL (2025-09-22 03:42:23 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:41:53.821360598 (565h33m29.174145726s) [upload=497735]: Starting upload Sep 22 03:41:53.821432342 (565h33m29.174217471s) [upload=497735]: Done starting upload Sep 22 03:41:53.821573697 (565h33m29.174358826s) [upload=497735]: Uploading segments... Sep 22 03:41:53.821895693 (565h33m29.174680821s) [upload=497735]: Got next segment. Inline: false Sep 22 03:41:53.821989369 (565h33m29.174774496s) [upload=497735]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:41:53.821995941 (565h33m29.174781069s) [upload=497735]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:41:53.843161018 (565h33m29.195946156s) [seg_pos={0 0}, upload=497735]: Begin upload segment... Sep 22 03:41:53.843219267 (565h33m29.196004405s) [seg_pos={0 0}, upload=497735]: Done begin upload segment. Sep 22 03:41:53.843260845 (565h33m29.196045973s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843270974 (565h33m29.196056102s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.843790710 (565h33m29.196575838s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844019850 (565h33m29.196804988s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844360168 (565h33m29.197145296s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844472439 (565h33m29.197257576s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844820201 (565h33m29.19760533s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.844922043 (565h33m29.197707171s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845271628 (565h33m29.198056767s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.845900639 (565h33m29.198685767s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846045771 (565h33m29.1988309s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.846725577 (565h33m29.199510705s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847449616 (565h33m29.200234744s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.847961646 (565h33m29.200746775s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.849104080 (565h33m29.201889208s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.850005882 (565h33m29.202791011s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Uploading piece... Sep 22 03:41:53.982635031 (565h33m29.335420168s) [upload=497735]: Next returned nil segment Sep 22 03:41:53.982652824 (565h33m29.335437952s) [upload=497735]: Waiting for error group managing segments... Sep 22 03:41:53.990044095 (565h33m29.342829232s) [noise=true , address=45.91.116.11:4118 , piece_id=R6HKZQ2B5AQ4C7YHHPBON4SVHPIV2IJN7DMSY6H7BSX7WHTSPKAQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.990531790 (565h33m29.343316918s) [noise=true , address=45.91.116.13:4154 , piece_id=M2Q6RNA3AKZUBYFE6OMIZOFTRX7ASU7D3SPCVUGUDLMIQ7TZIZWA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991047046 (565h33m29.343832184s) [noise=true , address=45.91.116.9:4261 , piece_id=KXFVZ2ES3MGT26OJRKEHYW5DPGPMEFUF2WTEZXEQFXZBLHEDILKA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991466263 (565h33m29.344251392s) [noise=true , address=45.91.116.7:4009 , piece_id=3K2HO6HF3KR7PIJRDABE7FW4A2FVEKHNR3ANPMU7MWVILDCKYX6Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.991769693 (565h33m29.344554831s) [noise=true , address=45.91.116.4:4939 , piece_id=X6PJQNIPP5DGU4LGUJZEXJXR7FAARPN77W5D6MBNRMHPDEPY25AA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992458005 (565h33m29.345243133s) [noise=true , address=77.90.31.19:4842 , piece_id=WOMDAGSW44HXQV7E6JQEQBZKHIYGLW25QPBRCPK2XCRT62MUAHYQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.992505003 (565h33m29.345290131s) [noise=true , address=45.91.116.7:4010 , piece_id=CKBCDXJE3ATNSEDNH4DNFH3WL4GZMVMEXVXRLI7GRY3XFUU2WOMA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993159922 (565h33m29.34594505s) [noise=true , address=77.90.31.12:4592 , piece_id=CKT6LHANOHVC4NHMRQ2SEEXFPQA7MHM44RKOISA45M26QAWIPDVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.993274718 (565h33m29.346059845s) [noise=true , address=45.91.116.8:4047 , piece_id=GWNXFH4LDIAYWDKNKVONYRFVTTKBQHLR72ZHHFZ7M2YDXCBXLRIQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994692838 (565h33m29.347477966s) [noise=true , address=77.90.31.21:4878 , piece_id=TPRVB7U4ZNR7BOYT4BC22JIDHV3IHBTWM76R25UK5JRYR4XUSI7A, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.994841236 (565h33m29.347626365s) [noise=true , address=77.90.31.13:4628 , piece_id=BLD55WPW4WBWJ6GWBBJDXWIYRQNPCUBA3N5UFZDT4WFKMDJOKP3Q, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:41:53.995361182 (565h33m29.348146311s) [noise=true , address=77.90.31.16:4733 , piece_id=MHKBKDVWJ2RZQXRLMTO33LSX7YJWJ3PMZJSVRO3RXKDXP2F7CSVA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: <nil> Sep 22 03:42:23.809279587 (565h33m59.162064714s) [noise=true , address=145.79.227.4:4186 , piece_id=DJEVDGISBPJJ4R37Z5RXWHVLOXZRPMMVC3PWO66H24JQHY46QAUA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809421593 (565h33m59.162206721s) [noise=true , address=145.79.227.12:4765 , piece_id=UJQMQKN3YUVQ2AQNXBPYV2ZUUKWY7Z4F6FHG4DXKI6ARMA2TFIXQ, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:42:23.809560264 (565h33m59.162345402s) [noise=true , address=145.79.227.6:4259 , piece_id=WORJPEIFUREEUUCOV3P4RPFEQLXGHSUD4ZBLNLN2OVJDUAHVTDHA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 03:42:23.809598205 (565h33m59.162383333s) [noise=true , address=145.79.227.18:4005 , piece_id=27PBR6OCJYCV76LXCWGZ4RMHYHYELU7UUQWK77ONLAKO7OZP5RJA, seg_pos={0 0}, upload=497735]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; context deadline exceeded Sep 22 03:42:23.809711137 (565h33m59.162496265s) [seg_pos={0 0}, upload=497735]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:42:23.825581299 (565h33m59.178366426s) [upload=497735]: Done uploading segments... Sep 22 03:42:23.825600274 (565h33m59.178385412s) [upload=497735]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.017901505s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X.......XX.................................................] duration: 2.200208831s FAIL (2025-09-22 03:42:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.200232095s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..........XX.....X..X...X..........X...X.XX...XXX...X......] duration: 461.118411ms FAIL (2025-09-22 03:42:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 461.146985ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX..........XX.....X..X...X..........X...X.XX...XXX...X.....] duration: 469.077607ms FAIL (2025-09-22 03:41:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 469.136467ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 4.938176841s FAIL (2025-09-22 03:41:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.938205214s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X....................................X..X...X...X.X.....X] duration: 30.015273347s FAIL (2025-09-22 03:41:31 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:41:01.871600412 (565h32m37.22438552s) [upload=497720]: Starting upload Sep 22 03:41:01.871653101 (565h32m37.224438199s) [upload=497720]: Done starting upload Sep 22 03:41:01.871730476 (565h32m37.224515574s) [upload=497720]: Uploading segments... Sep 22 03:41:01.871888362 (565h32m37.22467347s) [upload=497720]: Got next segment. Inline: false Sep 22 03:41:01.871934830 (565h32m37.224719938s) [upload=497720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:41:01.871938877 (565h32m37.224723975s) [upload=497720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:41:01.890319909 (565h32m37.243105018s) [seg_pos={0 0}, upload=497720]: Begin upload segment... Sep 22 03:41:01.890351910 (565h32m37.243137018s) [seg_pos={0 0}, upload=497720]: Done begin upload segment. Sep 22 03:41:01.890379401 (565h32m37.2431645s) [noise=true , address=77.90.31.17:4771 , piece_id=EFBGIIKAC6T5JJOQT5OECSZDXGGGHHIKD3PIOWZV75BIBWC2LRNQ, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.890385283 (565h32m37.24317038s) [noise=true , address=145.79.227.7:4293 , piece_id=XOBJLHJYT3NUY42QZAOWNXHOXTXPWOPTGSAFSIVEICVP4OBN36HA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.890683673 (565h32m37.243468781s) [noise=true , address=77.90.31.19:4842 , piece_id=BAIBGNJQXFGHVLGE4TGQ4SUDZMDBZ4HN7OJHUFDE7VLTDYLNKIEA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.890723799 (565h32m37.243508907s) [noise=true , address=45.91.116.9:4261 , piece_id=7TIM2TTXRIYHWNJWURKPN4NOX2YLUDC33EDERS5D5BI5EEXLKAZA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.890914015 (565h32m37.243699123s) [noise=true , address=45.91.116.10:4081 , piece_id=VATBBU2K7EJSXK5KVP3UV466IBU6R46YXXYW7EUAS4YCHASVSPZQ, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891017149 (565h32m37.243802246s) [noise=true , address=45.91.116.12:4227 , piece_id=7G6UIWRVEOZBROBOOYCZO3PKLNDRLQANDUH464WRNWOBFMWYF22Q, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891160618 (565h32m37.243945715s) [noise=true , address=77.90.31.13:4628 , piece_id=5EIQUAB7OOAT7THIOSZTP5Q624P2R7PT6XDHWMDBOQWDPAWVBR7Q, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891242742 (565h32m37.24402785s) [noise=true , address=145.79.227.6:4257 , piece_id=HP5DMAP6KXCTAT62YMFU6YQGU7RD2QOIBYG3OKR3C67WTKXPRRDA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891397312 (565h32m37.24418242s) [noise=true , address=145.79.227.2:4115 , piece_id=B3PL7B2XB36MNSCAHYEKRA5XHIQRBTNKOR3YPMTWOOQT2DCSKOFA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891471482 (565h32m37.244256579s) [noise=true , address=77.90.31.19:4843 , piece_id=XDZ53A3BYUO3AQOQIXI2MPOIJJND44TP45WQ2G5Q2NZX4DG7TCGA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891720729 (565h32m37.244505837s) [noise=true , address=45.91.116.6:4973 , piece_id=D2GYHIMWS2Z3UX6Y2KVH2AFDI4L3ONKC5PIAGZJH77ZKLSM3B7XQ, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.891981528 (565h32m37.244766636s) [noise=true , address=77.90.31.14:4662 , piece_id=LJDZOVSBBTQDIKMWQKDNZWF4PJ3D3V6OZ75DEC5CAKDRFFXLZGSA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.892226558 (565h32m37.245011656s) [noise=true , address=45.91.116.7:4010 , piece_id=N2ORNKABODGSLOAK3MTYRZ7SBZMJUTNVQWHNUBI4OMVTMYRWDUDQ, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.892444628 (565h32m37.245229736s) [noise=true , address=145.79.227.11:4730 , piece_id=OHYMTLDU7HT565ITPYSAWIW6TNGO337QUATZHGKPGG4YPFLOI4IQ, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.894227192 (565h32m37.2470123s) [noise=true , address=45.91.116.11:4117 , piece_id=4DJ55PYUAYQAJIWE5ACBRIU3SXB3UGTDNWZ2FWIYH5G45MRY4LSQ, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.894682617 (565h32m37.247467715s) [noise=true , address=45.91.116.5:4297 , piece_id=7CLUHX2RGSZV6BVFOIFTMKJ3QRKRA3G47KAJYD6PGTUX3WPC2GAA, seg_pos={0 0}, upload=497720]: Uploading piece... Sep 22 03:41:01.971552279 (565h32m37.324337376s) [upload=497720]: Next returned nil segment Sep 22 03:41:01.971565003 (565h32m37.324350111s) [upload=497720]: Waiting for error group managing segments... Sep 22 03:41:01.979626992 (565h32m37.3324121s) [noise=true , address=45.91.116.12:4227 , piece_id=7G6UIWRVEOZBROBOOYCZO3PKLNDRLQANDUH464WRNWOBFMWYF22Q, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.980663607 (565h32m37.333448705s) [noise=true , address=45.91.116.10:4081 , piece_id=VATBBU2K7EJSXK5KVP3UV466IBU6R46YXXYW7EUAS4YCHASVSPZQ, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.981025155 (565h32m37.333810263s) [noise=true , address=45.91.116.7:4010 , piece_id=N2ORNKABODGSLOAK3MTYRZ7SBZMJUTNVQWHNUBI4OMVTMYRWDUDQ, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.981356817 (565h32m37.334141925s) [noise=true , address=45.91.116.5:4297 , piece_id=7CLUHX2RGSZV6BVFOIFTMKJ3QRKRA3G47KAJYD6PGTUX3WPC2GAA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.981713808 (565h32m37.334498916s) [noise=true , address=45.91.116.11:4117 , piece_id=4DJ55PYUAYQAJIWE5ACBRIU3SXB3UGTDNWZ2FWIYH5G45MRY4LSQ, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.982296310 (565h32m37.335081418s) [noise=true , address=45.91.116.9:4261 , piece_id=7TIM2TTXRIYHWNJWURKPN4NOX2YLUDC33EDERS5D5BI5EEXLKAZA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.982611473 (565h32m37.335396581s) [noise=true , address=77.90.31.19:4842 , piece_id=BAIBGNJQXFGHVLGE4TGQ4SUDZMDBZ4HN7OJHUFDE7VLTDYLNKIEA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.983091703 (565h32m37.335876811s) [noise=true , address=77.90.31.17:4771 , piece_id=EFBGIIKAC6T5JJOQT5OECSZDXGGGHHIKD3PIOWZV75BIBWC2LRNQ, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.983409069 (565h32m37.336194177s) [noise=true , address=77.90.31.13:4628 , piece_id=5EIQUAB7OOAT7THIOSZTP5Q624P2R7PT6XDHWMDBOQWDPAWVBR7Q, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.983763253 (565h32m37.336548361s) [noise=true , address=77.90.31.19:4843 , piece_id=XDZ53A3BYUO3AQOQIXI2MPOIJJND44TP45WQ2G5Q2NZX4DG7TCGA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.984167782 (565h32m37.33695289s) [noise=true , address=45.91.116.6:4973 , piece_id=D2GYHIMWS2Z3UX6Y2KVH2AFDI4L3ONKC5PIAGZJH77ZKLSM3B7XQ, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:01.984372727 (565h32m37.337157835s) [noise=true , address=77.90.31.14:4662 , piece_id=LJDZOVSBBTQDIKMWQKDNZWF4PJ3D3V6OZ75DEC5CAKDRFFXLZGSA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: <nil> Sep 22 03:41:31.860976990 (565h33m7.213762129s) [noise=true , address=145.79.227.7:4293 , piece_id=XOBJLHJYT3NUY42QZAOWNXHOXTXPWOPTGSAFSIVEICVP4OBN36HA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:41:31.861026934 (565h33m7.213812062s) [noise=true , address=145.79.227.11:4730 , piece_id=OHYMTLDU7HT565ITPYSAWIW6TNGO337QUATZHGKPGG4YPFLOI4IQ, seg_pos={0 0}, upload=497720]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:41:31.861036973 (565h33m7.213822101s) [noise=true , address=145.79.227.2:4115 , piece_id=B3PL7B2XB36MNSCAHYEKRA5XHIQRBTNKOR3YPMTWOOQT2DCSKOFA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:41:31.861047242 (565h33m7.21383237s) [noise=true , address=145.79.227.6:4257 , piece_id=HP5DMAP6KXCTAT62YMFU6YQGU7RD2QOIBYG3OKR3C67WTKXPRRDA, seg_pos={0 0}, upload=497720]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:41:31.861102175 (565h33m7.213887304s) [seg_pos={0 0}, upload=497720]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:41:31.873967859 (565h33m7.226752987s) [upload=497720]: Done uploading segments... Sep 22 03:41:31.873983789 (565h33m7.226768917s) [upload=497720]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.015287193s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X.......XX................................................] duration: 2.18619079s FAIL (2025-09-22 03:41:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.18621716s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.306684651s FAIL (2025-09-22 03:41:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.306706332s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX..........XX.....X..X...X..........X...X.XX...XXX...X....] duration: 705.231178ms FAIL (2025-09-22 03:40:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 705.253159ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.054300696s FAIL (2025-09-22 03:40:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.054352113s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...X.......XX...............................................] duration: 2.158866688s FAIL (2025-09-22 03:40:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.158891144s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X....................................X..X...X...X.X.....] duration: 378.487169ms FAIL (2025-09-22 03:40:30 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:37:05.953774132 (565h28m41.30655924s) [upload=497661]: Starting upload Sep 22 03:37:05.953823264 (565h28m41.306608372s) [upload=497661]: Done starting upload Sep 22 03:37:05.953885071 (565h28m41.306670168s) [upload=497661]: Uploading segments... Sep 22 03:37:05.954033739 (565h28m41.306818837s) [upload=497661]: Got next segment. Inline: false Sep 22 03:37:05.954058936 (565h28m41.306844044s) [upload=497661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:37:05.954062213 (565h28m41.306847321s) [upload=497661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:37:05.978394584 (565h28m41.331179682s) [seg_pos={0 0}, upload=497661]: Begin upload segment... Sep 22 03:37:05.978424721 (565h28m41.331209829s) [seg_pos={0 0}, upload=497661]: Done begin upload segment. Sep 22 03:37:05.978443977 (565h28m41.331229085s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978484633 (565h28m41.331269741s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978762044 (565h28m41.331547153s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978783665 (565h28m41.331568773s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979021161 (565h28m41.331806269s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979040176 (565h28m41.331825274s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979239009 (565h28m41.332024117s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979346039 (565h28m41.332131147s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979977655 (565h28m41.332762763s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980195954 (565h28m41.332981053s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980470450 (565h28m41.333255558s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980850994 (565h28m41.333636092s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981390416 (565h28m41.334175524s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981549866 (565h28m41.334334974s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982114485 (565h28m41.334899593s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982508565 (565h28m41.335293663s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:06.088031568 (565h28m41.440816685s) [upload=497661]: Next returned nil segment Sep 22 03:37:06.088047888 (565h28m41.440832996s) [upload=497661]: Waiting for error group managing segments... Sep 22 03:37:06.097235449 (565h28m41.450020588s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098300347 (565h28m41.451085445s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098716739 (565h28m41.451501847s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098985253 (565h28m41.451770381s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099078658 (565h28m41.451863767s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099415269 (565h28m41.452200377s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099811354 (565h28m41.452596462s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099992072 (565h28m41.452777201s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100240468 (565h28m41.453025576s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100627725 (565h28m41.453412833s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100834132 (565h28m41.453619271s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.101022145 (565h28m41.453807254s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:35.942202163 (565h29m11.294987262s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942247639 (565h29m11.295032737s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942299857 (565h29m11.295084985s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942350993 (565h29m11.295136091s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942376942 (565h29m11.29516205s) [seg_pos={0 0}, upload=497661]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:37:35.957725684 (565h29m11.310510792s) [upload=497661]: Done uploading segments... Sep 22 03:37:35.957737848 (565h29m11.310522956s) [upload=497661]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 378.508199ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.155591917s FAIL (2025-09-22 03:40:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.155609259s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X....................................X..X...X...X.X....] duration: 522.529736ms FAIL (2025-09-22 03:40:05 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:37:05.953774132 (565h28m41.30655924s) [upload=497661]: Starting upload Sep 22 03:37:05.953823264 (565h28m41.306608372s) [upload=497661]: Done starting upload Sep 22 03:37:05.953885071 (565h28m41.306670168s) [upload=497661]: Uploading segments... Sep 22 03:37:05.954033739 (565h28m41.306818837s) [upload=497661]: Got next segment. Inline: false Sep 22 03:37:05.954058936 (565h28m41.306844044s) [upload=497661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:37:05.954062213 (565h28m41.306847321s) [upload=497661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:37:05.978394584 (565h28m41.331179682s) [seg_pos={0 0}, upload=497661]: Begin upload segment... Sep 22 03:37:05.978424721 (565h28m41.331209829s) [seg_pos={0 0}, upload=497661]: Done begin upload segment. Sep 22 03:37:05.978443977 (565h28m41.331229085s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978484633 (565h28m41.331269741s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978762044 (565h28m41.331547153s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978783665 (565h28m41.331568773s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979021161 (565h28m41.331806269s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979040176 (565h28m41.331825274s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979239009 (565h28m41.332024117s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979346039 (565h28m41.332131147s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979977655 (565h28m41.332762763s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980195954 (565h28m41.332981053s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980470450 (565h28m41.333255558s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980850994 (565h28m41.333636092s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981390416 (565h28m41.334175524s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981549866 (565h28m41.334334974s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982114485 (565h28m41.334899593s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982508565 (565h28m41.335293663s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:06.088031568 (565h28m41.440816685s) [upload=497661]: Next returned nil segment Sep 22 03:37:06.088047888 (565h28m41.440832996s) [upload=497661]: Waiting for error group managing segments... Sep 22 03:37:06.097235449 (565h28m41.450020588s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098300347 (565h28m41.451085445s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098716739 (565h28m41.451501847s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098985253 (565h28m41.451770381s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099078658 (565h28m41.451863767s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099415269 (565h28m41.452200377s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099811354 (565h28m41.452596462s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099992072 (565h28m41.452777201s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100240468 (565h28m41.453025576s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100627725 (565h28m41.453412833s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100834132 (565h28m41.453619271s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.101022145 (565h28m41.453807254s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:35.942202163 (565h29m11.294987262s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942247639 (565h29m11.295032737s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942299857 (565h29m11.295084985s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942350993 (565h29m11.295136091s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942376942 (565h29m11.29516205s) [seg_pos={0 0}, upload=497661]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:37:35.957725684 (565h29m11.310510792s) [upload=497661]: Done uploading segments... Sep 22 03:37:35.957737848 (565h29m11.310522956s) [upload=497661]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 522.560575ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X.......XX..............................................] duration: 2.313443937s FAIL (2025-09-22 03:39:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.313470266s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..XX..........XX.....X..X...X..........X...X.XX...XXX...X...] duration: 423.72487ms FAIL (2025-09-22 03:39:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 423.779432ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.007407117s FAIL (2025-09-22 03:39:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.007427365s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X....................................X..X...X...X.X...] duration: 400.93749ms FAIL (2025-09-22 03:39:17 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:37:05.953774132 (565h28m41.30655924s) [upload=497661]: Starting upload Sep 22 03:37:05.953823264 (565h28m41.306608372s) [upload=497661]: Done starting upload Sep 22 03:37:05.953885071 (565h28m41.306670168s) [upload=497661]: Uploading segments... Sep 22 03:37:05.954033739 (565h28m41.306818837s) [upload=497661]: Got next segment. Inline: false Sep 22 03:37:05.954058936 (565h28m41.306844044s) [upload=497661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:37:05.954062213 (565h28m41.306847321s) [upload=497661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:37:05.978394584 (565h28m41.331179682s) [seg_pos={0 0}, upload=497661]: Begin upload segment... Sep 22 03:37:05.978424721 (565h28m41.331209829s) [seg_pos={0 0}, upload=497661]: Done begin upload segment. Sep 22 03:37:05.978443977 (565h28m41.331229085s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978484633 (565h28m41.331269741s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978762044 (565h28m41.331547153s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978783665 (565h28m41.331568773s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979021161 (565h28m41.331806269s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979040176 (565h28m41.331825274s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979239009 (565h28m41.332024117s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979346039 (565h28m41.332131147s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979977655 (565h28m41.332762763s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980195954 (565h28m41.332981053s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980470450 (565h28m41.333255558s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980850994 (565h28m41.333636092s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981390416 (565h28m41.334175524s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981549866 (565h28m41.334334974s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982114485 (565h28m41.334899593s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982508565 (565h28m41.335293663s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:06.088031568 (565h28m41.440816685s) [upload=497661]: Next returned nil segment Sep 22 03:37:06.088047888 (565h28m41.440832996s) [upload=497661]: Waiting for error group managing segments... Sep 22 03:37:06.097235449 (565h28m41.450020588s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098300347 (565h28m41.451085445s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098716739 (565h28m41.451501847s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098985253 (565h28m41.451770381s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099078658 (565h28m41.451863767s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099415269 (565h28m41.452200377s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099811354 (565h28m41.452596462s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099992072 (565h28m41.452777201s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100240468 (565h28m41.453025576s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100627725 (565h28m41.453412833s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100834132 (565h28m41.453619271s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.101022145 (565h28m41.453807254s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:35.942202163 (565h29m11.294987262s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942247639 (565h29m11.295032737s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942299857 (565h29m11.295084985s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942350993 (565h29m11.295136091s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942376942 (565h29m11.29516205s) [seg_pos={0 0}, upload=497661]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:37:35.957725684 (565h29m11.310510792s) [upload=497661]: Done uploading segments... Sep 22 03:37:35.957737848 (565h29m11.310522956s) [upload=497661]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 400.955936ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...XX..........XX.....X..X...X..........X...X.XX...XXX...X..] duration: 433.867664ms FAIL (2025-09-22 03:39:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 433.884045ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 4.908807947s FAIL (2025-09-22 03:39:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.908831151s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......XX.............................................] duration: 2.188699483s FAIL (2025-09-22 03:38:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.188718638s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X....................................X..X...X...X.X..] duration: 473.50099ms FAIL (2025-09-22 03:38:45 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:37:05.953774132 (565h28m41.30655924s) [upload=497661]: Starting upload Sep 22 03:37:05.953823264 (565h28m41.306608372s) [upload=497661]: Done starting upload Sep 22 03:37:05.953885071 (565h28m41.306670168s) [upload=497661]: Uploading segments... Sep 22 03:37:05.954033739 (565h28m41.306818837s) [upload=497661]: Got next segment. Inline: false Sep 22 03:37:05.954058936 (565h28m41.306844044s) [upload=497661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:37:05.954062213 (565h28m41.306847321s) [upload=497661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:37:05.978394584 (565h28m41.331179682s) [seg_pos={0 0}, upload=497661]: Begin upload segment... Sep 22 03:37:05.978424721 (565h28m41.331209829s) [seg_pos={0 0}, upload=497661]: Done begin upload segment. Sep 22 03:37:05.978443977 (565h28m41.331229085s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978484633 (565h28m41.331269741s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978762044 (565h28m41.331547153s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978783665 (565h28m41.331568773s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979021161 (565h28m41.331806269s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979040176 (565h28m41.331825274s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979239009 (565h28m41.332024117s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979346039 (565h28m41.332131147s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979977655 (565h28m41.332762763s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980195954 (565h28m41.332981053s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980470450 (565h28m41.333255558s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980850994 (565h28m41.333636092s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981390416 (565h28m41.334175524s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981549866 (565h28m41.334334974s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982114485 (565h28m41.334899593s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982508565 (565h28m41.335293663s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:06.088031568 (565h28m41.440816685s) [upload=497661]: Next returned nil segment Sep 22 03:37:06.088047888 (565h28m41.440832996s) [upload=497661]: Waiting for error group managing segments... Sep 22 03:37:06.097235449 (565h28m41.450020588s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098300347 (565h28m41.451085445s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098716739 (565h28m41.451501847s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098985253 (565h28m41.451770381s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099078658 (565h28m41.451863767s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099415269 (565h28m41.452200377s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099811354 (565h28m41.452596462s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099992072 (565h28m41.452777201s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100240468 (565h28m41.453025576s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100627725 (565h28m41.453412833s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100834132 (565h28m41.453619271s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.101022145 (565h28m41.453807254s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:35.942202163 (565h29m11.294987262s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942247639 (565h29m11.295032737s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942299857 (565h29m11.295084985s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942350993 (565h29m11.295136091s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942376942 (565h29m11.29516205s) [seg_pos={0 0}, upload=497661]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:37:35.957725684 (565h29m11.310510792s) [upload=497661]: Done uploading segments... Sep 22 03:37:35.957737848 (565h29m11.310522956s) [upload=497661]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 473.52719ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 476.236933ms FAIL (2025-09-22 03:38:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 476.259455ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....XX..........XX.....X..X...X..........X...X.XX...XXX...X.] duration: 404.97247ms FAIL (2025-09-22 03:38:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 405.077057ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.152958615s FAIL (2025-09-22 03:38:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.15298245s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....XX..........XX.....X..X...X..........X...X.XX...XXX...X] duration: 30.000799932s FAIL (2025-09-22 03:38:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00082506s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [......X.......XX............................................] duration: 2.419912929s FAIL (2025-09-22 03:37:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.419933207s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 542.054865ms FAIL (2025-09-22 03:37:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 542.077368ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X....................................X..X...X...X.X.] duration: 444.446174ms FAIL (2025-09-22 03:37:41 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:37:05.953774132 (565h28m41.30655924s) [upload=497661]: Starting upload Sep 22 03:37:05.953823264 (565h28m41.306608372s) [upload=497661]: Done starting upload Sep 22 03:37:05.953885071 (565h28m41.306670168s) [upload=497661]: Uploading segments... Sep 22 03:37:05.954033739 (565h28m41.306818837s) [upload=497661]: Got next segment. Inline: false Sep 22 03:37:05.954058936 (565h28m41.306844044s) [upload=497661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:37:05.954062213 (565h28m41.306847321s) [upload=497661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:37:05.978394584 (565h28m41.331179682s) [seg_pos={0 0}, upload=497661]: Begin upload segment... Sep 22 03:37:05.978424721 (565h28m41.331209829s) [seg_pos={0 0}, upload=497661]: Done begin upload segment. Sep 22 03:37:05.978443977 (565h28m41.331229085s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978484633 (565h28m41.331269741s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978762044 (565h28m41.331547153s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978783665 (565h28m41.331568773s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979021161 (565h28m41.331806269s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979040176 (565h28m41.331825274s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979239009 (565h28m41.332024117s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979346039 (565h28m41.332131147s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979977655 (565h28m41.332762763s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980195954 (565h28m41.332981053s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980470450 (565h28m41.333255558s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980850994 (565h28m41.333636092s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981390416 (565h28m41.334175524s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981549866 (565h28m41.334334974s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982114485 (565h28m41.334899593s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982508565 (565h28m41.335293663s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:06.088031568 (565h28m41.440816685s) [upload=497661]: Next returned nil segment Sep 22 03:37:06.088047888 (565h28m41.440832996s) [upload=497661]: Waiting for error group managing segments... Sep 22 03:37:06.097235449 (565h28m41.450020588s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098300347 (565h28m41.451085445s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098716739 (565h28m41.451501847s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098985253 (565h28m41.451770381s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099078658 (565h28m41.451863767s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099415269 (565h28m41.452200377s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099811354 (565h28m41.452596462s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099992072 (565h28m41.452777201s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100240468 (565h28m41.453025576s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100627725 (565h28m41.453412833s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100834132 (565h28m41.453619271s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.101022145 (565h28m41.453807254s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:35.942202163 (565h29m11.294987262s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942247639 (565h29m11.295032737s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942299857 (565h29m11.295084985s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942350993 (565h29m11.295136091s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942376942 (565h29m11.29516205s) [seg_pos={0 0}, upload=497661]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:37:35.957725684 (565h29m11.310510792s) [upload=497661]: Done uploading segments... Sep 22 03:37:35.957737848 (565h29m11.310522956s) [upload=497661]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.472213ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.499012483s FAIL (2025-09-22 03:37:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.499066314s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 730.585677ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:37:37 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 730.608038ms core satellite ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X........X....................................X..X...X...X.X] duration: 30.017656149s FAIL (2025-09-22 03:37:35 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:37:05.953774132 (565h28m41.30655924s) [upload=497661]: Starting upload Sep 22 03:37:05.953823264 (565h28m41.306608372s) [upload=497661]: Done starting upload Sep 22 03:37:05.953885071 (565h28m41.306670168s) [upload=497661]: Uploading segments... Sep 22 03:37:05.954033739 (565h28m41.306818837s) [upload=497661]: Got next segment. Inline: false Sep 22 03:37:05.954058936 (565h28m41.306844044s) [upload=497661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:37:05.954062213 (565h28m41.306847321s) [upload=497661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:37:05.978394584 (565h28m41.331179682s) [seg_pos={0 0}, upload=497661]: Begin upload segment... Sep 22 03:37:05.978424721 (565h28m41.331209829s) [seg_pos={0 0}, upload=497661]: Done begin upload segment. Sep 22 03:37:05.978443977 (565h28m41.331229085s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978484633 (565h28m41.331269741s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978762044 (565h28m41.331547153s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.978783665 (565h28m41.331568773s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979021161 (565h28m41.331806269s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979040176 (565h28m41.331825274s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979239009 (565h28m41.332024117s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979346039 (565h28m41.332131147s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.979977655 (565h28m41.332762763s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980195954 (565h28m41.332981053s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980470450 (565h28m41.333255558s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.980850994 (565h28m41.333636092s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981390416 (565h28m41.334175524s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.981549866 (565h28m41.334334974s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982114485 (565h28m41.334899593s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:05.982508565 (565h28m41.335293663s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Uploading piece... Sep 22 03:37:06.088031568 (565h28m41.440816685s) [upload=497661]: Next returned nil segment Sep 22 03:37:06.088047888 (565h28m41.440832996s) [upload=497661]: Waiting for error group managing segments... Sep 22 03:37:06.097235449 (565h28m41.450020588s) [noise=true , address=45.91.116.12:4227 , piece_id=NJSYZC3KI2JSO4JNAHEAVPZOM4SXJQSGP2RLUP42AJA2AMHP3FOQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098300347 (565h28m41.451085445s) [noise=true , address=45.91.116.14:4191 , piece_id=PELOJMBKOOL6HTJSN67RVVBO6EARPEJYZ2AOHATWNAU2G4LKQ63A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098716739 (565h28m41.451501847s) [noise=true , address=45.91.116.7:4010 , piece_id=BTVWKA43RBSVLA7C4P64GQVXKJNDPVOUKNJHM5IQK5RKYUGVRZXA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.098985253 (565h28m41.451770381s) [noise=true , address=45.91.116.13:4154 , piece_id=ATWLJYV7JJOPBXH5FHGMBZF4262MQATB3SQHRKZEXLOJGYSTXO3A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099078658 (565h28m41.451863767s) [noise=true , address=45.91.116.14:4190 , piece_id=JGNODIHDLDUNQBGTERYG2HEALMVR6C2DNJIW5DJ53SL6BEGYQ5QQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099415269 (565h28m41.452200377s) [noise=true , address=45.91.116.9:4261 , piece_id=WLD6HOS74N7KEQP5K5PASGNJERFUQKS43IRLJGZKAENQD7BMZJZQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099811354 (565h28m41.452596462s) [noise=true , address=45.91.116.4:4938 , piece_id=336YLDIDEUPV6AY7QHUIRH752NXWJ2XQPJHMYATFMZQHHSYGHJ7A, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.099992072 (565h28m41.452777201s) [noise=true , address=45.91.116.5:4298 , piece_id=FUALPJISAK67FN6MLJLS6AZZYPIU4IHQZ3KA4S4TII7FPBPO6WMA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100240468 (565h28m41.453025576s) [noise=true , address=45.91.116.11:4117 , piece_id=U3UAIS7FEHOXHLBCMEOZSQXUCKBVYLFO6BKT6EHZDUEKZPIKSRBQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100627725 (565h28m41.453412833s) [noise=true , address=77.90.31.15:4697 , piece_id=KK6ZHFRPQKS2CZTDDP6UQRCBKPXD2RXONIXYBG4I3E57LHTNSJ3Q, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.100834132 (565h28m41.453619271s) [noise=true , address=77.90.31.13:4628 , piece_id=YJMXUDB5QM2CZNEIJQ7NMAZUYD26S6ZULBGUT4QUWVVQPDLJ67OA, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:06.101022145 (565h28m41.453807254s) [noise=true , address=77.90.31.14:4662 , piece_id=322PF3Y7RD3HRR7PVT4VJCHRSWTXWI2MPQ5GDPTOBFAJEQZSUVIQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: <nil> Sep 22 03:37:35.942202163 (565h29m11.294987262s) [noise=true , address=145.79.227.11:4730 , piece_id=UTMC45NTSXWAR5XVXK2TUXKZQU5R7HZSWDM4L3BSY6AL2QB4WJFQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942247639 (565h29m11.295032737s) [noise=true , address=145.79.227.4:4185 , piece_id=EIARGJHN7VYMPE7CWUL3OUJ4LVI7KZ5CM4WPIUQPCIWHH6UH7OKQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: rpc: dial tcp 145.79.227.4:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942299857 (565h29m11.295084985s) [noise=true , address=145.79.227.18:4005 , piece_id=E6QVRWIWFR27URNYAGBHTEMQWF4GGJ3W2F376PQMS4GUIO6VJ2CQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942350993 (565h29m11.295136091s) [noise=true , address=145.79.227.3:4149 , piece_id=MXTBHJDVFNLVZW4NL4HQ6GE6F2UXPJ6XQC2DFT4ZO4VK5A3O44SQ, seg_pos={0 0}, upload=497661]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:37:35.942376942 (565h29m11.29516205s) [seg_pos={0 0}, upload=497661]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:37:35.957725684 (565h29m11.310510792s) [upload=497661]: Done uploading segments... Sep 22 03:37:35.957737848 (565h29m11.310522956s) [upload=497661]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.01767212s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......X.......XX...........................................] duration: 2.221394195s FAIL (2025-09-22 03:37:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.221414403s core uplink ap1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)