Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX.XX.XX......XX..X....XX.XXXX.XX..X...X........X..........] duration: 688.577692ms FAIL (2025-09-22 03:05:09 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 688.605855ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X.......XX....] duration: 2.218124091s FAIL (2025-09-22 03:04:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.218143066s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 755.108001ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:04:47 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 755.125874ms core satellite ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X...X.....X.XX..XX..X.........X...X.......XX.........] duration: 627.985136ms FAIL (2025-09-22 03:04:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 628.042252ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 425.667871ms FAIL (2025-09-22 03:04:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 425.696134ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 824.977061ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:04:30 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 824.995096ms core satellite ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.192127621s FAIL (2025-09-22 03:04:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.192148219s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.......X...X.....X.XX..XX..X.........X...X.......XX........] duration: 456.538943ms FAIL (2025-09-22 03:04:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.551066ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.XX.XX.XX......XX..X....XX.XXXX.XX..X...X........X.........] duration: 415.850597ms FAIL (2025-09-22 03:04:01 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 415.875373ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 429.684468ms FAIL (2025-09-22 03:03:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 429.712361ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X.......XX...] duration: 2.417593241s FAIL (2025-09-22 03:03:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.417618027s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 774.402631ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:03:50 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 774.426745ms core satellite ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.XX.XX.XX......XX..X....XX.XXXX.XX..X...X........X........] duration: 448.795872ms FAIL (2025-09-22 03:03:22 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 448.818455ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.......X...X.....X.XX..XX..X.........X...X.......XX.......] duration: 605.305716ms FAIL (2025-09-22 03:03:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 605.330542ms edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 4.961785454s FAIL (2025-09-22 03:03:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.961806444s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [................................................X.......XX..] duration: 2.155793953s FAIL (2025-09-22 03:03:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.155823217s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 856.324272ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:03:07 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 856.347695ms core satellite ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 458.13665ms FAIL (2025-09-22 03:03:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 458.157388ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.338814958s FAIL (2025-09-22 03:02:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.338848241s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 56.43832259s FAIL (2025-09-22 03:02:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 56.438386261s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.XX.XX.XX......XX..X....XX.XXXX.XX..X...X........X.......] duration: 595.224267ms FAIL (2025-09-22 03:02:21 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 595.263661ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 505.558894ms FAIL (2025-09-22 03:02:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 505.580073ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X.......XX.] duration: 2.205316368s FAIL (2025-09-22 03:02:12 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.205330514s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X.......XX] duration: 30.000548317s FAIL (2025-09-22 03:02:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000559218s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 4.105842528s target: https://ap1.storj.io/login FAIL (2025-09-22 03:02:10 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 4.105868608s core satellite ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.......X...X.....X.XX..XX..X.........X...X.......XX......] duration: 388.670712ms FAIL (2025-09-22 03:02:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 388.689898ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X.XX.XX.XX......XX..X....XX.XXXX.XX..X...X........X......] duration: 603.224018ms FAIL (2025-09-22 03:01:44 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 603.253754ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.......X] duration: 30.000242306s FAIL (2025-09-22 03:01:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 30.000267442s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 7.011462216s target: https://ap1.storj.io/login FAIL (2025-09-22 03:01:37 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 7.011516257s core satellite ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 370.107437ms FAIL (2025-09-22 03:01:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 370.123037ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 423.282874ms FAIL (2025-09-22 03:00:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 423.308933ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.......X...X.....X.XX..XX..X.........X...X.......XX.....] duration: 505.235435ms FAIL (2025-09-22 03:00:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 505.452952ms edge gateway us-select-1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 724.703064ms target: https://ap1.storj.io/login FAIL (2025-09-22 03:00:51 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 724.722459ms core satellite ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX..X.XX.XX.XX......XX..X....XX.XXXX.XX..X...X........X.....] duration: 423.719012ms FAIL (2025-09-22 03:00:47 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 423.755001ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.22512068s FAIL (2025-09-22 03:00:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.225151667s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X.......X...X.....X.XX..XX..X.........X...X.......XX....] duration: 480.445365ms FAIL (2025-09-22 03:00:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 480.469349ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 538.267322ms FAIL (2025-09-22 03:00:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 538.289585ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX..X.XX.XX.XX......XX..X....XX.XXXX.XX..X...X........X....] duration: 392.599687ms FAIL (2025-09-22 03:00:00 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.619583ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.150656243s FAIL (2025-09-22 02:59:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.150677443s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 725.239231ms target: https://ap1.storj.io/login FAIL (2025-09-22 02:59:43 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 725.26075ms core satellite ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X.......X...X.....X.XX..XX..X.........X...X.......XX...] duration: 391.775148ms FAIL (2025-09-22 02:59:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 391.796508ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXX..X.XX.XX.XX......XX..X....XX.XXXX.XX..X...X........X...] duration: 569.434911ms FAIL (2025-09-22 02:59:25 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 569.452985ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 504.800358ms FAIL (2025-09-22 02:59:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 504.823331ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......X.......X...X.....X.XX..XX..X.........X...X.......XX..] duration: 412.901373ms FAIL (2025-09-22 02:59:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 412.913646ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 394.671774ms FAIL (2025-09-22 02:59:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=497068]: Starting upload Sep 22 02:57:19.602369921 (564h48m54.955155029s) [upload=497068]: Done starting upload Sep 22 02:57:19.602469568 (564h48m54.955254677s) [upload=497068]: Uploading segments... Sep 22 02:57:19.602726330 (564h48m54.955511428s) [upload=497068]: Got next segment. Inline: false Sep 22 02:57:19.602774871 (564h48m54.955559969s) [upload=497068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:19.602780101 (564h48m54.955565219s) [upload=497068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:19.645724990 (564h48m54.998510098s) [seg_pos={0 0}, upload=497068]: Begin upload segment... Sep 22 02:57:19.645790614 (564h48m54.998575712s) [seg_pos={0 0}, upload=497068]: Done begin upload segment. Sep 22 02:57:19.645811793 (564h48m54.998596901s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.645831420 (564h48m54.998616518s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646098541 (564h48m54.998883639s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646186856 (564h48m54.998971955s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646321099 (564h48m54.999106198s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646554938 (564h48m54.999340036s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646622525 (564h48m54.999407623s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.646776654 (564h48m54.999561762s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647042442 (564h48m54.999827551s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647366281 (564h48m55.000151389s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647642008 (564h48m55.000427106s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647873091 (564h48m55.000658189s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.647939065 (564h48m55.000724163s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.648291717 (564h48m55.001076816s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652209910 (564h48m55.004995018s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652433619 (564h48m55.005218727s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652681976 (564h48m55.005467073s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.652918220 (564h48m55.005703328s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Uploading piece... Sep 22 02:57:19.722974015 (564h48m55.075759113s) [upload=497068]: Next returned nil segment Sep 22 02:57:19.722987761 (564h48m55.075772859s) [upload=497068]: Waiting for error group managing segments... Sep 22 02:57:19.731288267 (564h48m55.084073375s) [noise=true , address=145.79.227.46:4609 , piece_id=2ZUOLR2THIRT2B5U33MCEJTMHOEYIA63X7PJZWODGRKUZY333POQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.731879286 (564h48m55.084664384s) [noise=true , address=145.79.227.35:4488 , piece_id=FVQ7LCFUW7AMPGQ6BBAJLMFWB3V4PL4NPHU2IBJM7W4S4CDH5A6A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.732663838 (564h48m55.085448946s) [noise=true , address=77.90.31.11:4564 , piece_id=EERPDLL4KY7H5AWILCN7SLOZEUPYMJBDDD7H5Z4R2RWQE7Y3J6MQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733610835 (564h48m55.086395943s) [noise=true , address=145.79.227.25:4636 , piece_id=55F5Q4724LMPWA4GGE5VPYII367G25Q6UBEB42JYHSOITRD34IHA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.733956455 (564h48m55.086741552s) [noise=true , address=145.79.227.47:4277 , piece_id=STF2XIBDA3QQF7VBQO2XZ63XHVMKEZDSIWEGTDJP4LBCW3GYV2CA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734772986 (564h48m55.087558083s) [noise=true , address=145.79.227.33:4902 , piece_id=EJRE4ABO52GW2ZOREEMXL4UKCQ6UB62QH5TMJI5BG7B4NZSF6CIA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.734820946 (564h48m55.087606054s) [noise=true , address=145.79.227.28:4759 , piece_id=ACXUEBF5FM5WTYQOJ5XSBT67WA357TEDL72WTOQZJK3SHA7WJCBQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.736621885 (564h48m55.089406983s) [noise=true , address=145.79.227.49:4364 , piece_id=QJTIZXY4KJQZ72KKZRN2QJNZIVMBIVXV3GNZUNJM6FZUL56ATSFQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.744527511 (564h48m55.097312619s) [noise=true , address=145.79.227.49:4351 , piece_id=FXOFGSDT57XT4C4N5KZMV7SZILQMTL75KZAM5GBYR4VC2OUATNUQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.753478278 (564h48m55.106263377s) [noise=true , address=145.79.227.24:4598 , piece_id=FARPW6UUM54BSRUXVJUE5C5IV4FAEZN2XKZ6YVA5GVKBPH6FOGGA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.757789318 (564h48m55.110574426s) [noise=true , address=145.79.227.34:4917 , piece_id=BJGXDSOVIZVS2TENTCMJESSHRE5M5BS5TARN3WUHI5QMY7CVWD7Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.763604260 (564h48m55.116389358s) [noise=true , address=145.79.227.22:4515 , piece_id=P7OLTUWXJQDDGDOMXQ5BYJSSBTIB2FD5SYVS6UQUNNSYHZ2OAQRA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:57:19.773135106 (564h48m55.125920214s) [noise=true , address=145.79.227.43:4161 , piece_id=YZ3FNK66XJQSMMEGDN636IIMDICOFSJX53OETT3VE2Y2KCIOVG7A, seg_pos={0 0}, upload=497068]: Done uploading piece. err: <nil> Sep 22 02:58:19.587557404 (564h49m54.940342522s) [noise=true , address=145.79.227.13:4813 , piece_id=TGMSYHJOCDKN5JYL6STLSU7TKRU7JTKJWOBPJU5ICSWKXH6ZLOAQ, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587651280 (564h49m54.940436398s) [noise=true , address=145.79.227.3:4183 , piece_id=L25HWLZAULCEFT5YGHPEPZUCKKTQ4GUVKW3CR5OKULC4XOZGVKBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:145.79.227.3:4183): protocol: rpc: dial tcp 145.79.227.3:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587681085 (564h49m54.940466214s) [noise=true , address=145.79.227.5:4237 , piece_id=WM64JAKN2AUFVG357SA2TF55QEN6PR27MUFQX7BLUDMBD44EC25Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:145.79.227.5:4237): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587750837 (564h49m54.940535964s) [noise=true , address=145.79.227.12:4793 , piece_id=G63ZJRQ4O5X32LFAM7P3T64KKNST77IAKXM3O46QUK42O5NKUQBA, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:145.79.227.12:4793): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:58:19.587844532 (564h49m54.94062966s) [noise=true , address=145.79.227.13:4821 , piece_id=6RSGVCHJHEJPCYLFGYFNQHHSNSL7I565DHOPGIYK5GGRQQLFM57Q, seg_pos={0 0}, upload=497068]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:145.79.227.13:4821): protocol: rpc: dial tcp 145.79.227.13:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:58:19.587902271 (564h49m54.940687439s) [seg_pos={0 0}, upload=497068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 22 02:58:19.609127882 (564h49m54.961913011s) [upload=497068]: Done uploading segments... Sep 22 02:58:19.609146166 (564h49m54.961931295s) [upload=497068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 394.694196ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.187228922s FAIL (2025-09-22 02:59:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.187262366s core uplink ap1
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 751.8185ms target: https://ap1.storj.io/login FAIL (2025-09-22 02:58:54 UTC) orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout Duration: 751.857723ms core satellite ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXX..X.XX.XX.XX......XX..X....XX.XXXX.XX..X...X........X..] duration: 446.481099ms FAIL (2025-09-22 02:58:36 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:57:08.281940234 (564h48m43.634725342s) [upload=497064]: Starting upload Sep 22 02:57:08.281995938 (564h48m43.634781046s) [upload=497064]: Done starting upload Sep 22 02:57:08.282071449 (564h48m43.634856578s) [upload=497064]: Uploading segments... Sep 22 02:57:08.282236871 (564h48m43.635021979s) [upload=497064]: Got next segment. Inline: false Sep 22 02:57:08.282274551 (564h48m43.63505968s) [upload=497064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:57:08.282281825 (564h48m43.635066953s) [upload=497064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:57:08.301081935 (564h48m43.653867063s) [seg_pos={0 0}, upload=497064]: Begin upload segment... Sep 22 02:57:08.301119636 (564h48m43.653904764s) [seg_pos={0 0}, upload=497064]: Done begin upload segment. Sep 22 02:57:08.301152938 (564h48m43.653938046s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301171593 (564h48m43.653956721s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301475353 (564h48m43.654260461s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301658757 (564h48m43.654443885s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301723969 (564h48m43.654509077s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.301945896 (564h48m43.654730995s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302047447 (564h48m43.654832575s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302173934 (564h48m43.654959042s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302400249 (564h48m43.655185346s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302498714 (564h48m43.655283851s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302619369 (564h48m43.655404477s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.302934452 (564h48m43.65571958s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303299276 (564h48m43.656084414s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.303783665 (564h48m43.656568794s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304260449 (564h48m43.657045578s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.304649841 (564h48m43.657434969s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Uploading piece... Sep 22 02:57:08.408885477 (564h48m43.761670574s) [upload=497064]: Next returned nil segment Sep 22 02:57:08.408896908 (564h48m43.761682006s) [upload=497064]: Waiting for error group managing segments... Sep 22 02:57:08.417535079 (564h48m43.770320177s) [noise=true , address=45.91.116.14:4189 , piece_id=HTW6U45MRHEXKNR35DUY374KMNVKPNHRJV4K5TOXW2EI53COT5PQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.417865899 (564h48m43.770650998s) [noise=true , address=45.91.116.12:4225 , piece_id=3UHKLHTRTFTOPDDM5QGGKQYO6H3KTSPNAYUV2YBZGGWBOPVLEUUQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418178976 (564h48m43.770964105s) [noise=true , address=45.91.116.12:4227 , piece_id=RR3PDGKUTP3WCP3SYB7WRYEYPTMCRXIGLVFXMX5GMTPB6JNRCVZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418365918 (564h48m43.771151057s) [noise=true , address=45.91.116.9:4262 , piece_id=BHF7N2IOY4QEXS2JGNJMQ2MSLTVFTXROPLLST62HSJKPECW6QH5Q, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418472909 (564h48m43.771258007s) [noise=true , address=77.90.31.18:4806 , piece_id=Z67HG7K7CSZCUMKR2UFWMDKM75DAXJ3P62ZZKMWEO4PM2G3IBDZQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.418832323 (564h48m43.771617421s) [noise=true , address=45.91.116.8:4047 , piece_id=56WTV2CNWFYGX2QHT25LCPHMQH32VZEGETFW2YRJVR2VZVI6IDHQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419246100 (564h48m43.772031198s) [noise=true , address=45.91.116.11:4119 , piece_id=CQT53UNDD6GB3BRQPMPGXO36A2C3GUWHTWIBEYYHERLW3GOI55VQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419414676 (564h48m43.772199814s) [noise=true , address=45.91.116.6:4974 , piece_id=MKHRQQDRD5BRKLWU6VCERUF2ZMEGXK7ORBPGXZJJD3GRQ5JESPBQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.419533740 (564h48m43.772318837s) [noise=true , address=45.91.116.4:4937 , piece_id=UATS2IK3YMOGIM3AIX6LC32REBX654P7SE6CGVLQFPIBGCCEKPMQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420666545 (564h48m43.773451683s) [noise=true , address=45.91.116.13:4155 , piece_id=ABEANYFL6VZQD56OEL7XFLBWO7SXV6SC2ICBBDGPTSNGAWANBDRA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.420852824 (564h48m43.773637923s) [noise=true , address=77.90.31.19:4842 , piece_id=URJOWGORIFGZDMH2OLKUQWAFBJKYD54HHZ2S6J3RENWI3LZHGGDQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:08.421253537 (564h48m43.774038665s) [noise=true , address=77.90.31.13:4627 , piece_id=U6N7WUE52AKOQJGQG2XPMOQQSWOPKJP4A62D6GEUDVUTFODDKMZA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: <nil> Sep 22 02:57:38.269181747 (564h49m13.621966875s) [noise=true , address=145.79.227.11:4730 , piece_id=NIT6NGSYDU5NTHIZTZPUUV5YOF6Z2T3EWS6PEWQYDGIMJWYA5RMA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269239155 (564h49m13.622024283s) [noise=true , address=145.79.227.19:4077 , piece_id=N52U3LQ2N2YCPUGJRRXEYELXMQQ6F463WGK6HK2UJM6C6MFNCPFA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269287445 (564h49m13.622072573s) [noise=true , address=145.79.227.14:4838 , piece_id=HUEUHZ4O64WZ255NBVPCQKK6SWHZQMLXW2BP2AB7CZNTCJ55JLGQ, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:57:38.269309677 (564h49m13.622094805s) [noise=true , address=145.79.227.17:4970 , piece_id=G2NWG5IUI2PXNQGUZ5TGRGQ6TIOOJA75C7XDZU3PLPNC6CMVBCBA, seg_pos={0 0}, upload=497064]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: rpc: dial tcp 145.79.227.17:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:57:38.269333642 (564h49m13.62211878s) [seg_pos={0 0}, upload=497064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:57:38.288426071 (564h49m13.641211209s) [upload=497064]: Done uploading segments... Sep 22 02:57:38.288464884 (564h49m13.641250012s) [upload=497064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 446.502889ms core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.366041391s FAIL (2025-09-22 02:58:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.366063733s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X.......X...X.....X.XX..XX..X.........X...X.......XX.] duration: 393.556991ms FAIL (2025-09-22 02:58:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 393.573032ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)