Orbiter History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 4.970199664s FAIL (2025-09-22 03:22:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.970221414s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 428.11608ms FAIL (2025-09-22 03:22:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 428.144935ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.........X...X.......XX..........XX.....X..X...X.........] duration: 678.533492ms FAIL (2025-09-22 03:22:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 678.556104ms edge gateway us-select-1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 763.268511ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:22:01 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 763.293448ms core satellite ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.......XX.........................] duration: 2.340653023s FAIL (2025-09-22 03:21:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.340676077s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..XX.XXXX.XX..X...X........X................................] duration: 521.322921ms FAIL (2025-09-22 03:21:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 521.383334ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 4.932441697s FAIL (2025-09-22 03:21:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.932475401s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 743.98015ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:21:32 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 743.997184ms core satellite ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 452.121499ms FAIL (2025-09-22 03:21:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 452.145524ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..X.........X...X.......XX..........XX.....X..X...X........] duration: 465.345253ms FAIL (2025-09-22 03:21:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 465.363657ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.......XX........................] duration: 2.615532401s FAIL (2025-09-22 03:20:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.615559362s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX..X.........X...X.......XX..........XX.....X..X...X.......] duration: 666.057485ms FAIL (2025-09-22 03:20:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 666.084196ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 425.318411ms FAIL (2025-09-22 03:20:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 425.346864ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.04318817s FAIL (2025-09-22 03:20:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.043237093s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 747.928731ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:20:33 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 747.963415ms core satellite ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX..X.........X...X.......XX..........XX.....X..X...X......] duration: 367.54264ms FAIL (2025-09-22 03:20:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 367.666503ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...XX.XXXX.XX..X...X........X...............................] duration: 398.110343ms FAIL (2025-09-22 03:20:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 398.133695ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 470.766608ms FAIL (2025-09-22 03:20:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 470.786554ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.......XX.......................] duration: 2.232062332s FAIL (2025-09-22 03:19:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.232084725s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..XX..X.........X...X.......XX..........XX.....X..X...X.....] duration: 425.057539ms FAIL (2025-09-22 03:19:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 425.077866ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 621.573566ms FAIL (2025-09-22 03:19:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 621.59128ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.037731398s FAIL (2025-09-22 03:19:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.037750835s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....XX.XXXX.XX..X...X........X..............................] duration: 415.049831ms FAIL (2025-09-22 03:19:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 415.071571ms core uplink us-select-1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 721.425716ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:19:26 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 721.458948ms core satellite ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 367.541677ms FAIL (2025-09-22 03:19:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 367.560804ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 4.845887152s FAIL (2025-09-22 03:19:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.845908612s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 744.268565ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:18:52 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 744.284465ms core satellite ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [............................X.......XX......................] duration: 2.205810459s FAIL (2025-09-22 03:18:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.205828293s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X..XX..X.........X...X.......XX..........XX.....X..X...X....] duration: 464.621395ms FAIL (2025-09-22 03:18:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 464.643676ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X....XX.XXXX.XX..X...X........X.............................] duration: 377.072193ms FAIL (2025-09-22 03:18:37 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 377.090537ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 488.499721ms FAIL (2025-09-22 03:18:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 488.528303ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 4.975507283s FAIL (2025-09-22 03:18:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.975540535s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX..XX..X.........X...X.......XX..........XX.....X..X...X...] duration: 705.766182ms FAIL (2025-09-22 03:17:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 705.786701ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X....XX.XXXX.XX..X...X........X............................] duration: 802.758924ms FAIL (2025-09-22 03:17:44 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 802.779902ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.......XX.....................] duration: 2.170631808s FAIL (2025-09-22 03:17:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.17065926s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 759.943158ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:17:42 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 759.965791ms core satellite ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XX..XX..X.........X...X.......XX..........XX.....X..X...X..] duration: 359.733688ms FAIL (2025-09-22 03:17:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 359.769344ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 446.890401ms FAIL (2025-09-22 03:17:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 446.913476ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 824.309667ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:17:26 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 824.334142ms core satellite ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X....XX.XXXX.XX..X...X........X...........................] duration: 462.471612ms FAIL (2025-09-22 03:17:16 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 462.492311ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.035045073s FAIL (2025-09-22 03:16:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.035071292s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 751.236678ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:16:49 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 751.260973ms core satellite ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.......XX....................] duration: 2.221869095s FAIL (2025-09-22 03:16:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.221895815s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 810.789247ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:16:33 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 810.803914ms core satellite ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.XX..XX..X.........X...X.......XX..........XX.....X..X...X.] duration: 375.87229ms FAIL (2025-09-22 03:16:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 375.89886ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X..X....XX.XXXX.XX..X...X........X..........................] duration: 397.008295ms FAIL (2025-09-22 03:16:27 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 397.033462ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 431.58972ms FAIL (2025-09-22 03:16:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 431.612723ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.104082083s FAIL (2025-09-22 03:16:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.104105778s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.XX..XX..X.........X...X.......XX..........XX.....X..X...X] duration: 30.000580783s FAIL (2025-09-22 03:16:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000601633s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.......XX...................] duration: 2.214886563s FAIL (2025-09-22 03:16:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.214910219s core uplink ap1

Orbiter Check History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)