Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 13.149686ms FAIL (2025-08-07 22:52:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 13.166086ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 22 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXX.XXXXX.X.............................] duration: 187.949903ms FAIL (2025-08-07 22:51:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 420501s) [upload=6055]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:06.462569787 (5h55m20.049626008s) [upload=6061]: Starting upload Aug 7 22:22:06.462655577 (5h55m20.049711788s) [upload=6061]: Done starting upload Aug 7 22:22:06.462750936 (5h55m20.049807157s) [upload=6061]: Uploading segments... Aug 7 22:22:06.462782464 (5h55m20.049838675s) [upload=6061]: Got next segment. Inline: true Aug 7 22:22:06.462786181 (5h55m20.049842402s) [upload=6061]: Waiting for error group managing segments... Aug 7 22:22:06.462817791 (5h55m20.049874001s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:06.462821648 (5h55m20.049877869s) [upload=6061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:06.462823982 (5h55m20.049880203s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:36.449657257 (5h55m50.036713498s) [upload=6061]: Done uploading segments... Aug 7 22:22:36.449678196 (5h55m50.036734448s) [upload=6061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:59.911739779 (5h56m13.498795989s) [upload=6078]: Starting upload Aug 7 22:22:59.911786736 (5h56m13.498842957s) [upload=6078]: Done starting upload Aug 7 22:22:59.911827472 (5h56m13.498883683s) [upload=6078]: Uploading segments... Aug 7 22:22:59.911861074 (5h56m13.498917295s) [upload=6078]: Got next segment. Inline: true Aug 7 22:22:59.911864401 (5h56m13.498920622s) [upload=6078]: Waiting for error group managing segments... Aug 7 22:22:59.911889107 (5h56m13.498945319s) [upload=6078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:59.911893005 (5h56m13.498949226s) [upload=6078]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:59.911894898 (5h56m13.498951119s) [upload=6078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:29.900458747 (5h56m43.487514958s) [upload=6078]: Done uploading segments... Aug 7 22:23:29.900473835 (5h56m43.487530047s) [upload=6078]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:58.370463254 (5h57m11.957519465s) [upload=6094]: Starting upload Aug 7 22:23:58.370512907 (5h57m11.957569128s) [upload=6094]: Done starting upload Aug 7 22:23:58.370553031 (5h57m11.957609243s) [upload=6094]: Uploading segments... Aug 7 22:23:58.370578639 (5h57m11.95763486s) [upload=6094]: Got next segment. Inline: true Aug 7 22:23:58.370582427 (5h57m11.957638648s) [upload=6094]: Waiting for error group managing segments... Aug 7 22:23:58.370610951 (5h57m11.957667172s) [upload=6094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:58.370615178 (5h57m11.957671399s) [upload=6094]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:58.370617392 (5h57m11.957673613s) [upload=6094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:58.450972793 (5h57m12.038029004s) [upload=6094]: Done uploading segments... Aug 7 22:23:58.450986078 (5h57m12.038042289s) [upload=6094]: Upload finished. err: <nil> Aug 7 22:24:40.794473113 (5h57m54.381529325s) [upload=6107]: Starting upload Aug 7 22:24:40.794523428 (5h57m54.381579639s) [upload=6107]: Done starting upload Aug 7 22:24:40.794565206 (5h57m54.381621417s) [upload=6107]: Uploading segments... Aug 7 22:24:40.794591815 (5h57m54.381648026s) [upload=6107]: Got next segment. Inline: true Aug 7 22:24:40.794595622 (5h57m54.381651833s) [upload=6107]: Waiting for error group managing segments... Aug 7 22:24:40.794624367 (5h57m54.381680577s) [upload=6107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:40.794628323 (5h57m54.381684535s) [upload=6107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:40.794630638 (5h57m54.381686849s) [upload=6107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:10.774549311 (5h58m24.361605533s) [upload=6107]: Done uploading segments... Aug 7 22:25:10.774570019 (5h58m24.361626231s) [upload=6107]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:10.855666099 (5h58m24.442722339s) [upload=6114]: Starting upload Aug 7 22:25:10.855752680 (5h58m24.442808921s) [upload=6114]: Done starting upload Aug 7 22:25:10.855822740 (5h58m24.442878991s) [upload=6114]: Uploading segments... Aug 7 22:25:10.855866683 (5h58m24.442922923s) [upload=6114]: Got next segment. Inline: true Aug 7 22:25:10.855872564 (5h58m24.442928805s) [upload=6114]: Waiting for error group managing segments... Aug 7 22:25:10.855966008 (5h58m24.443022249s) [upload=6114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:10.855972971 (5h58m24.443029212s) [upload=6114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:10.855976568 (5h58m24.443032819s) [upload=6114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:40.842457902 (5h58m54.429514153s) [upload=6114]: Done uploading segments... Aug 7 22:25:40.842479021 (5h58m54.429535272s) [upload=6114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:40.940807525 (5h58m54.527863766s) [upload=6122]: Starting upload Aug 7 22:25:40.940896992 (5h58m54.527953233s) [upload=6122]: Done starting upload Aug 7 22:25:40.940964388 (5h58m54.528020628s) [upload=6122]: Uploading segments... Aug 7 22:25:40.941010434 (5h58m54.528066675s) [upload=6122]: Got next segment. Inline: true Aug 7 22:25:40.941016375 (5h58m54.528072616s) [upload=6122]: Waiting for error group managing segments... Aug 7 22:25:40.941060899 (5h58m54.52811715s) [upload=6122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:40.941068091 (5h58m54.528124332s) [upload=6122]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:40.941073060 (5h58m54.528129311s) [upload=6122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:10.929528068 (5h59m24.516584319s) [upload=6122]: Done uploading segments... Aug 7 22:26:10.929548557 (5h59m24.516604797s) [upload=6122]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:11.008231659 (5h59m24.59528788s) [upload=6129]: Starting upload Aug 7 22:26:11.008286501 (5h59m24.595342713s) [upload=6129]: Done starting upload Aug 7 22:26:11.008333198 (5h59m24.595389419s) [upload=6129]: Uploading segments... Aug 7 22:26:11.008364838 (5h59m24.595421059s) [upload=6129]: Got next segment. Inline: true Aug 7 22:26:11.008368595 (5h59m24.595424816s) [upload=6129]: Waiting for error group managing segments... Aug 7 22:26:11.008399943 (5h59m24.595456164s) [upload=6129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:11.008403599 (5h59m24.59545981s) [upload=6129]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:11.008406565 (5h59m24.595462786s) [upload=6129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:40.996305184 (5h59m54.583361385s) [upload=6129]: Done uploading segments... Aug 7 22:26:40.996323769 (5h59m54.58337998s) [upload=6129]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:14.829073707 (6h0m28.416129948s) [upload=6144]: Starting upload Aug 7 22:27:14.829153145 (6h0m28.416209386s) [upload=6144]: Done starting upload Aug 7 22:27:14.829247050 (6h0m28.416303291s) [upload=6144]: Uploading segments... Aug 7 22:27:14.829299899 (6h0m28.41635614s) [upload=6144]: Got next segment. Inline: true Aug 7 22:27:14.829306702 (6h0m28.416362943s) [upload=6144]: Waiting for error group managing segments... Aug 7 22:27:14.829350864 (6h0m28.416407105s) [upload=6144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:14.829357416 (6h0m28.416413657s) [upload=6144]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:14.829361143 (6h0m28.416417384s) [upload=6144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:44.808549878 (6h0m58.395606099s) [upload=6144]: Done uploading segments... Aug 7 22:27:44.808564877 (6h0m58.395621098s) [upload=6144]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:23.619901331 (6h1m37.206957573s) [upload=6162]: Starting upload Aug 7 22:28:23.620017008 (6h1m37.207073248s) [upload=6162]: Done starting upload Aug 7 22:28:23.620072551 (6h1m37.207128791s) [upload=6162]: Uploading segments... Aug 7 22:28:23.620125059 (6h1m37.2071813s) [upload=6162]: Got next segment. Inline: true Aug 7 22:28:23.620132432 (6h1m37.207188672s) [upload=6162]: Waiting for error group managing segments... Aug 7 22:28:23.620198335 (6h1m37.207254577s) [upload=6162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:23.620204407 (6h1m37.207260647s) [upload=6162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:23.620208153 (6h1m37.207264395s) [upload=6162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:23.683041871 (6h1m37.270098112s) [upload=6162]: Done uploading segments... Aug 7 22:28:23.683055537 (6h1m37.270111778s) [upload=6162]: Upload finished. err: <nil> Aug 7 22:28:40.804037302 (6h1m54.391093523s) [upload=6165]: Starting upload Aug 7 22:28:40.804089460 (6h1m54.391145671s) [upload=6165]: Done starting upload Aug 7 22:28:40.804134294 (6h1m54.391190515s) [upload=6165]: Uploading segments... Aug 7 22:28:40.804181422 (6h1m54.391237632s) [upload=6165]: Got next segment. Inline: true Aug 7 22:28:40.804185379 (6h1m54.3912416s) [upload=6165]: Waiting for error group managing segments... Aug 7 22:28:40.804217198 (6h1m54.391273419s) [upload=6165]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:40.804220915 (6h1m54.391277127s) [upload=6165]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:40.804223370 (6h1m54.391279591s) [upload=6165]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:10.793781791 (6h2m24.380838023s) [upload=6165]: Done uploading segments... Aug 7 22:29:10.793803392 (6h2m24.380859603s) [upload=6165]: Upload finished. err: metaclient: context deadline exceeded Duration: 187.971481ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXX..XXXX.X.............................] duration: 423.370043ms FAIL (2025-08-07 22:51:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 423.388097ms edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 113.187048ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:51:53 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 113.20877ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXX...............................] duration: 1.164412921s FAIL (2025-08-07 22:51:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=6089]: Done starting upload Aug 7 22:23:43.961837605 (5h56m57.548893826s) [upload=6089]: Uploading segments... Aug 7 22:23:43.962138106 (5h56m57.549194347s) [upload=6089]: Got next segment. Inline: false Aug 7 22:23:43.962204581 (5h56m57.549260822s) [upload=6089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:43.962209980 (5h56m57.549266222s) [upload=6089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:12.375269699 (5h57m25.96232592s) [seg_pos={0 0}, upload=6089]: Begin upload segment... Aug 7 22:24:12.375313250 (5h57m25.962369471s) [seg_pos={0 0}, upload=6089]: Done begin upload segment. Aug 7 22:24:12.375339780 (5h57m25.962396031s) [noise=true , address=193.149.164.143:4698 , piece_id=AV2DCKYWWT5N4PL57WTXW45JNFYOCRLLEY6YSBSWOAQAPHAVXJQQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.375356050 (5h57m25.962412261s) [noise=true , address=104.36.82.53:4156 , piece_id=NBWZ4FB4S7CTDN2OTWTDKSH2R7DJN66P7GRTQTQ5FJZFQCZLDAQQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.375715481 (5h57m25.962771693s) [noise=true , address=104.36.82.58:4280 , piece_id=W2A65H4FIUOCUSMSVCANR2OPVBUFO4YNAFZV3KCASSCSPCHONGBA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.375724769 (5h57m25.96278099s) [noise=true , address=104.36.82.59:4333 , piece_id=JONL6XDWEUYMHVDPJVB54PE5HCYWN7KRO3FSNAMPJWCEP6QBKYVQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376022925 (5h57m25.963079146s) [noise=true , address=45.91.116.9:4270 , piece_id=CMJ7K7XDV44UUOWXU6NOUNMWZGLOEZKPTLVFMTNWIR6OK52TOEKQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376075263 (5h57m25.963131475s) [noise=true , address=104.36.82.36:4652 , piece_id=ZDJ2LSAX25MBAAW4PW745SX7J7X3S4PAGXEUS36T43KTM3LZ2WRQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376490167 (5h57m25.963546388s) [noise=true , address=104.36.82.45:4921 , piece_id=DCEBY7YTNAO4UR6YZ5LC5W27INFXZQ26BPJ7BHPMXY26P5BDL23Q, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376527327 (5h57m25.963583599s) [noise=true , address=104.36.82.12:4169 , piece_id=XPNXIWBPTKCEEQTVQ3BAQ3HCLQE5U5BTRIAR3FQ6O4OQJZMCJ6ZA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376755913 (5h57m25.963812125s) [noise=true , address=104.36.82.62:4411 , piece_id=3TMITFZMXF5ESOCJNLFU6EIT7CFUYULYZ6FJUQ6X7T2WBEQMKWFQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376864006 (5h57m25.963920216s) [noise=true , address=104.36.82.14:4243 , piece_id=WKT5D56ZXVSJJHVR3GB6N5XR6CGU4VW2XK2YTMPBZOH4S2ZPOVCA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376979641 (5h57m25.964035862s) [noise=true , address=104.36.82.9:4031 , piece_id=UOVVYSA77DQOK53ATDDXV56PLEVS6GBQ5HMZEKCEURYBPRCXNYPA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377222374 (5h57m25.964278595s) [noise=true , address=104.36.82.52:4100 , piece_id=7RUPILCVG3GL3KB4CS25YCYKDB4C6HFT6TGZWZIDC5EHBI2SD55Q, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377246499 (5h57m25.96430271s) [noise=true , address=79.127.201.215:4343 , piece_id=N3KGRRGXWJLCJGYSLPOLOIYZHP7AFWV6VBCHVBETDYOQBIXIWPMQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377620267 (5h57m25.964676488s) [noise=true , address=104.36.82.35:4597 , piece_id=6UMSPJQOVLEDWMIQMLAJLZ2AY6Y5XLMODCN3V63EB5JNJS3EC7BA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377642138 (5h57m25.964698349s) [noise=true , address=104.36.82.16:4300 , piece_id=RJRU7SETYXWMQK7GEE3HBD2XMEVSI3UJ6ZDCIWVW4266QSNAOPZA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377932059 (5h57m25.96498828s) [noise=true , address=104.36.82.47:4544 , piece_id=DVQRASYHRKPFEONZEWJ3MBVM7O5764NPWYADLMG4ARAH5PBD55MA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377950764 (5h57m25.965006985s) [noise=true , address=104.36.82.21:4478 , piece_id=5UO756WUOKCS52UJI2NIQFUSU7W6NBW34MGFH6YZX27HKWY5243A, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.378280479 (5h57m25.9653367s) [noise=true , address=104.36.82.37:4656 , piece_id=A25QOY4UFT7RRRY6PVAS3NNML7PGJM4OHDH56PF2HZHKJ7VOPURA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.468526485 (5h57m26.055582696s) [upload=6089]: Next returned nil segment Aug 7 22:24:12.468535081 (5h57m26.055591292s) [upload=6089]: Waiting for error group managing segments... Aug 7 22:24:12.483383738 (5h57m26.070439949s) [noise=true , address=104.36.82.37:4656 , piece_id=A25QOY4UFT7RRRY6PVAS3NNML7PGJM4OHDH56PF2HZHKJ7VOPURA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.500821799 (5h57m26.08787801s) [noise=true , address=104.36.82.16:4300 , piece_id=RJRU7SETYXWMQK7GEE3HBD2XMEVSI3UJ6ZDCIWVW4266QSNAOPZA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.504410257 (5h57m26.091466458s) [noise=true , address=45.91.116.9:4270 , piece_id=CMJ7K7XDV44UUOWXU6NOUNMWZGLOEZKPTLVFMTNWIR6OK52TOEKQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.511708921 (5h57m26.098765132s) [noise=true , address=104.36.82.35:4597 , piece_id=6UMSPJQOVLEDWMIQMLAJLZ2AY6Y5XLMODCN3V63EB5JNJS3EC7BA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.523758022 (5h57m26.110814233s) [noise=true , address=104.36.82.12:4169 , piece_id=XPNXIWBPTKCEEQTVQ3BAQ3HCLQE5U5BTRIAR3FQ6O4OQJZMCJ6ZA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.533614250 (5h57m26.120670461s) [noise=true , address=104.36.82.52:4100 , piece_id=7RUPILCVG3GL3KB4CS25YCYKDB4C6HFT6TGZWZIDC5EHBI2SD55Q, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.557032764 (5h57m26.144088985s) [noise=true , address=104.36.82.21:4478 , piece_id=5UO756WUOKCS52UJI2NIQFUSU7W6NBW34MGFH6YZX27HKWY5243A, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.582698011 (5h57m26.169754223s) [noise=true , address=104.36.82.53:4156 , piece_id=NBWZ4FB4S7CTDN2OTWTDKSH2R7DJN66P7GRTQTQ5FJZFQCZLDAQQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.609343038 (5h57m26.196399279s) [noise=true , address=79.127.201.215:4343 , piece_id=N3KGRRGXWJLCJGYSLPOLOIYZHP7AFWV6VBCHVBETDYOQBIXIWPMQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.639134328 (5h57m26.226190538s) [noise=true , address=104.36.82.36:4652 , piece_id=ZDJ2LSAX25MBAAW4PW745SX7J7X3S4PAGXEUS36T43KTM3LZ2WRQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.956879031 (5h57m26.543935272s) [noise=true , address=193.149.164.143:4698 , piece_id=AV2DCKYWWT5N4PL57WTXW45JNFYOCRLLEY6YSBSWOAQAPHAVXJQQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:13.036416979 (5h57m26.623473231s) [noise=true , address=104.36.82.9:4031 , piece_id=UOVVYSA77DQOK53ATDDXV56PLEVS6GBQ5HMZEKCEURYBPRCXNYPA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:13.172901316 (5h57m26.759957557s) [noise=true , address=104.36.82.47:4544 , piece_id=DVQRASYHRKPFEONZEWJ3MBVM7O5764NPWYADLMG4ARAH5PBD55MA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:13.202754351 (5h57m26.789810592s) [noise=true , address=104.36.82.14:4243 , piece_id=WKT5D56ZXVSJJHVR3GB6N5XR6CGU4VW2XK2YTMPBZOH4S2ZPOVCA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:13.202797742 (5h57m26.789853984s) [seg_pos={0 0}, upload=6089]: Segment reached optimal threshold of 14 pieces. Aug 7 22:24:13.203007333 (5h57m26.790063585s) [noise=true , address=104.36.82.58:4280 , piece_id=W2A65H4FIUOCUSMSVCANR2OPVBUFO4YNAFZV3KCASSCSPCHONGBA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fgHaEFKK4CeGg4x3vywTuYfT3kmac31xqg2QpsahQNXmvjhW8): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:13.203190164 (5h57m26.790246415s) [noise=true , address=104.36.82.59:4333 , piece_id=JONL6XDWEUYMHVDPJVB54PE5HCYWN7KRO3FSNAMPJWCEP6QBKYVQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K7U6NXcB6SFBtQFrG7DXt2skACr3u5DX2QsrnToi8EKywS9TN): protocol: write tcp 5.161.89.252:63730->104.36.82.59:4333: use of closed network connection; write tcp 5.161.89.252:63730->104.36.82.59:4333: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63730->104.36.82.59:4333: use of closed network connection Aug 7 22:24:13.203221153 (5h57m26.790277394s) [noise=true , address=104.36.82.45:4921 , piece_id=DCEBY7YTNAO4UR6YZ5LC5W27INFXZQ26BPJ7BHPMXY26P5BDL23Q, seg_pos={0 0}, upload=6089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bBRKXiuSN2cw6KrdP8q4Dsf4MiFCUSgMEGjqG2dKL9RLTQra1): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:13.203424783 (5h57m26.790481023s) [noise=true , address=104.36.82.62:4411 , piece_id=3TMITFZMXF5ESOCJNLFU6EIT7CFUYULYZ6FJUQ6X7T2WBEQMKWFQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: context canceled; context canceled Aug 7 22:24:13.203452675 (5h57m26.790508916s) [seg_pos={0 0}, upload=6089]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:24:13.203591695 (5h57m26.790647935s) [upload=6089]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:13.203602313 (5h57m26.790658565s) [upload=6089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:03.852878645 (6h2m17.439934866s) [upload=6089]: Done uploading segments... Aug 7 22:29:03.852904053 (6h2m17.439960294s) [upload=6089]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.16443955s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXX..X..............................] duration: 1.489890019s FAIL (2025-08-07 22:51:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.489942658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 154.938653ms FAIL (2025-08-07 22:51:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 154.954433ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXX.XXX..................................] duration: 349.927573ms FAIL (2025-08-07 22:51:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== TVRWLFHTOF2YZFDYX3LU5FENJAS6JOW572GLJ2OHVQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.749054955 (5h56m13.336111176s) [noise=true , address=104.36.82.2:4730 , piece_id=FKFZNBUC7L2WVTTFPNRSPJUWW6HAEE6CNUAYSHVDYXW5XENS462Q, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.749426248 (5h56m13.336482459s) [noise=true , address=45.91.116.9:4262 , piece_id=R7J2NVZJTXR7ATH2VUC75B2Z73TSJAQUBYJ3WE75RQ6AD3L3PSCQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.750681361 (5h56m13.337737582s) [noise=true , address=104.36.82.5:4840 , piece_id=6SYENOMYMIEYCEFSHJKMXOQE4RCDPDWBAHYDVRTXCQYWJGQDCCKA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.751093369 (5h56m13.338149581s) [noise=true , address=104.36.82.14:4221 , piece_id=SBKEVRBOXXBFOVTZWPSDVDPXEDV6HZIVAV4MWFIYDMNEU643KBWA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.751643326 (5h56m13.338699547s) [noise=true , address=45.91.116.4:4940 , piece_id=UXKWVJQC6KYUHGXKHRYHJATOP5KQF4BY2LWNQJQEGFYKQEBT6GWA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.752249137 (5h56m13.339305348s) [noise=true , address=77.90.31.13:4628 , piece_id=BYQWTXNUD76XIWZZAUZNHDUX7F7CABMDCPK7M7QPDHS3CFBRBMVQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.752826465 (5h56m13.339882676s) [noise=true , address=104.36.82.2:4731 , piece_id=L6W56WDLHKGHEPXBK7F2OYZ6CRM66ZVKG5RGJPH3FHC7RZZ3QGBA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.753473372 (5h56m13.340529583s) [noise=true , address=45.91.116.12:4227 , piece_id=7RIB2VCODFSHTRTBGI3Z74IECO3ATQXSXW7CY3TCA43MZKSKEQYA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.753961884 (5h56m13.341018096s) [noise=true , address=77.90.31.10:4519 , piece_id=QF2SVPCWB5HG4S7GQL6NFHTYPVOHP4HHORRY2TMJSRXP5A3J43TQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.754616727 (5h56m13.341672948s) [noise=true , address=104.36.82.9:4006 , piece_id=MDXFW3M6TBQJXYHMBTPIOD5I5PK3NUHDRZ3EERNCRKFVMUPSW43Q, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.755122451 (5h56m13.342178672s) [noise=true , address=104.36.82.8:4971 , piece_id=2MZZBGEPYJSHOBAG4FUAU5NKKAOL72C2AWD3GZECQXGPMOG6TDMQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.837536198 (5h56m13.424592419s) [upload=6077]: Next returned nil segment Aug 7 22:22:59.837547418 (5h56m13.424603629s) [upload=6077]: Waiting for error group managing segments... Aug 7 22:22:59.846810959 (5h56m13.43386718s) [noise=true , address=45.91.116.7:4011 , piece_id=E7YGVXQXIO2A34B22AHGKBQXER4KLEX3AFGHVFUQ6USLQMUFSW2Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.847279324 (5h56m13.434335535s) [noise=true , address=45.91.116.12:4227 , piece_id=7RIB2VCODFSHTRTBGI3Z74IECO3ATQXSXW7CY3TCA43MZKSKEQYA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.848559454 (5h56m13.435615675s) [noise=true , address=45.91.116.4:4940 , piece_id=UXKWVJQC6KYUHGXKHRYHJATOP5KQF4BY2LWNQJQEGFYKQEBT6GWA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.852382881 (5h56m13.439439102s) [noise=true , address=45.91.116.9:4262 , piece_id=R7J2NVZJTXR7ATH2VUC75B2Z73TSJAQUBYJ3WE75RQ6AD3L3PSCQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.852577974 (5h56m13.439634195s) [noise=true , address=77.90.31.19:4842 , piece_id=ULCHXICRO3ULX7IEW6GVXHHGXT6F42N3WQDEEUVOSKCXID6F6XZQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.856642091 (5h56m13.443698302s) [noise=true , address=104.36.82.8:4971 , piece_id=2MZZBGEPYJSHOBAG4FUAU5NKKAOL72C2AWD3GZECQXGPMOG6TDMQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.856766534 (5h56m13.443822745s) [noise=true , address=77.90.31.10:4519 , piece_id=QF2SVPCWB5HG4S7GQL6NFHTYPVOHP4HHORRY2TMJSRXP5A3J43TQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.868085922 (5h56m13.455142143s) [noise=true , address=45.91.116.12:4225 , piece_id=MNJGBICBIG3HQXCO7B2C2ITRFSOW475RAGNVTAQNVFQYAJ7DOU5Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.868246762 (5h56m13.455302973s) [noise=true , address=104.36.82.2:4730 , piece_id=FKFZNBUC7L2WVTTFPNRSPJUWW6HAEE6CNUAYSHVDYXW5XENS462Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.877804453 (5h56m13.464860674s) [noise=true , address=104.36.82.11:4115 , piece_id=BALIRX6S5WW2CGSXSIFHWV2ABDH3ASPHR36YEYRUYTTGZCW6U6AQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.882357252 (5h56m13.469413462s) [noise=true , address=104.36.82.2:4731 , piece_id=L6W56WDLHKGHEPXBK7F2OYZ6CRM66ZVKG5RGJPH3FHC7RZZ3QGBA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.887398722 (5h56m13.474454933s) [noise=true , address=104.36.82.14:4221 , piece_id=SBKEVRBOXXBFOVTZWPSDVDPXEDV6HZIVAV4MWFIYDMNEU643KBWA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.887674137 (5h56m13.474730358s) [noise=true , address=104.36.82.10:4079 , piece_id=DHLJEMGMQUTVRWLFHTOF2YZFDYX3LU5FENJAS6JOW572GLJ2OHVQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.887689665 (5h56m13.474745877s) [seg_pos={0 0}, upload=6077]: Segment reached optimal threshold of 13 pieces. Aug 7 22:22:59.887775385 (5h56m13.474831606s) [noise=true , address=77.90.31.13:4628 , piece_id=BYQWTXNUD76XIWZZAUZNHDUX7F7CABMDCPK7M7QPDHS3CFBRBMVQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Aug 7 22:22:59.887795453 (5h56m13.474851674s) [noise=true , address=104.36.82.5:4840 , piece_id=6SYENOMYMIEYCEFSHJKMXOQE4RCDPDWBAHYDVRTXCQYWJGQDCCKA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Aug 7 22:22:59.887879469 (5h56m13.47493569s) [noise=true , address=104.36.82.9:4006 , piece_id=MDXFW3M6TBQJXYHMBTPIOD5I5PK3NUHDRZ3EERNCRKFVMUPSW43Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 5.161.89.252:59712->104.36.82.9:4006: use of closed network connection; write tcp 5.161.89.252:59712->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:59712->104.36.82.9:4006: use of closed network connection Aug 7 22:22:59.887943669 (5h56m13.47499988s) [seg_pos={0 0}, upload=6077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:22:59.888006888 (5h56m13.475063109s) [upload=6077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:22:59.888010665 (5h56m13.475066886s) [upload=6077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:59.933305033 (5h56m13.520361243s) [upload=6077]: Done uploading segments... Aug 7 22:22:59.933317746 (5h56m13.520373958s) [upload=6077]: Upload finished. err: <nil> Aug 7 22:23:47.739190890 (5h57m1.326247131s) [upload=6091]: Starting upload Aug 7 22:23:47.739297178 (5h57m1.326353419s) [upload=6091]: Done starting upload Aug 7 22:23:47.739428053 (5h57m1.326484274s) [upload=6091]: Uploading segments... Aug 7 22:23:47.739777785 (5h57m1.326834025s) [upload=6091]: Got next segment. Inline: false Aug 7 22:23:47.739805196 (5h57m1.326861407s) [upload=6091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:47.739811638 (5h57m1.326867849s) [upload=6091]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:17.722308743 (5h57m31.309364985s) [upload=6091]: Next segment err: context deadline exceeded Aug 7 22:24:17.722438996 (5h57m31.309495238s) [upload=6091]: Done uploading segments... Aug 7 22:24:17.722476005 (5h57m31.309532246s) [upload=6091]: Upload finished. err: context deadline exceeded Aug 7 22:24:18.676382471 (5h57m32.263438711s) [upload=6101]: Starting upload Aug 7 22:24:18.676482739 (5h57m32.263538979s) [upload=6101]: Done starting upload Aug 7 22:24:18.676619723 (5h57m32.263675964s) [upload=6101]: Uploading segments... Aug 7 22:24:18.676886642 (5h57m32.263942883s) [upload=6101]: Got next segment. Inline: false Aug 7 22:24:18.676922940 (5h57m32.263979181s) [upload=6101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:18.676930835 (5h57m32.263987076s) [upload=6101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:48.662510024 (5h58m2.249566255s) [upload=6101]: Next segment err: context deadline exceeded Aug 7 22:24:48.662708235 (5h58m2.249764465s) [upload=6101]: Done uploading segments... Aug 7 22:24:48.662735284 (5h58m2.249791526s) [upload=6101]: Upload finished. err: context deadline exceeded Aug 7 22:25:18.142081018 (5h58m31.729137259s) [upload=6117]: Starting upload Aug 7 22:25:18.142160586 (5h58m31.729216828s) [upload=6117]: Done starting upload Aug 7 22:25:18.142286381 (5h58m31.729342623s) [upload=6117]: Uploading segments... Aug 7 22:25:18.142594587 (5h58m31.729650828s) [upload=6117]: Got next segment. Inline: false Aug 7 22:25:18.142641995 (5h58m31.729698236s) [upload=6117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:18.142647436 (5h58m31.729703676s) [upload=6117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:25:48.125202589 (5h59m1.71225884s) [upload=6117]: Next segment err: context deadline exceeded Aug 7 22:25:48.125308446 (5h59m1.712364688s) [upload=6117]: Done uploading segments... Aug 7 22:25:48.125355273 (5h59m1.712411525s) [upload=6117]: Upload finished. err: context deadline exceeded Duration: 349.950325ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 14 out of 60 is failed (threshold=3) [.........XXXXXXXXXXX.X.XX...................................] duration: 215.912067ms FAIL (2025-08-07 22:51:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:04.377299713 (5h52m17.964355924s) [upload=6018]: Starting upload Aug 7 22:19:04.377350166 (5h52m17.964406387s) [upload=6018]: Done starting upload Aug 7 22:19:04.377401523 (5h52m17.964457744s) [upload=6018]: Uploading segments... Aug 7 22:19:04.377446897 (5h52m17.964503118s) [upload=6018]: Got next segment. Inline: true Aug 7 22:19:04.377453410 (5h52m17.96450962s) [upload=6018]: Waiting for error group managing segments... Aug 7 22:19:04.377487142 (5h52m17.964543354s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:04.377491361 (5h52m17.964547582s) [upload=6018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:04.377493875 (5h52m17.964550086s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:04.361964803 (5h53m17.949021024s) [upload=6018]: Done uploading segments... Aug 7 22:20:04.361979240 (5h53m17.949035461s) [upload=6018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:11.847632631 (5h53m25.434688843s) [upload=6035]: Starting upload Aug 7 22:20:11.847694537 (5h53m25.434750747s) [upload=6035]: Done starting upload Aug 7 22:20:11.847740192 (5h53m25.434796403s) [upload=6035]: Uploading segments... Aug 7 22:20:11.847772231 (5h53m25.434828442s) [upload=6035]: Got next segment. Inline: true Aug 7 22:20:11.847776860 (5h53m25.434833071s) [upload=6035]: Waiting for error group managing segments... Aug 7 22:20:11.847824679 (5h53m25.434880891s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:11.847830130 (5h53m25.434886341s) [upload=6035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:11.847833205 (5h53m25.434889417s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:11.836000640 (5h54m25.42305688s) [upload=6035]: Done uploading segments... Aug 7 22:21:11.836026087 (5h54m25.423082328s) [upload=6035]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:11.965120248 (5h54m25.552176489s) [upload=6050]: Starting upload Aug 7 22:21:11.965231515 (5h54m25.552287757s) [upload=6050]: Done starting upload Aug 7 22:21:11.965301156 (5h54m25.552357397s) [upload=6050]: Uploading segments... Aug 7 22:21:11.965348053 (5h54m25.552404304s) [upload=6050]: Got next segment. Inline: true Aug 7 22:21:11.965353994 (5h54m25.552410235s) [upload=6050]: Waiting for error group managing segments... Aug 7 22:21:11.965399049 (5h54m25.55245529s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:11.965405170 (5h54m25.552461411s) [upload=6050]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:11.965408957 (5h54m25.552465198s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:11.952333050 (5h55m25.539389261s) [upload=6050]: Done uploading segments... Aug 7 22:22:11.952348108 (5h55m25.539404329s) [upload=6050]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:12.125652873 (5h55m25.712709094s) [upload=6063]: Starting upload Aug 7 22:22:12.125702476 (5h55m25.712758688s) [upload=6063]: Done starting upload Aug 7 22:22:12.125747020 (5h55m25.712803241s) [upload=6063]: Uploading segments... Aug 7 22:22:12.125776895 (5h55m25.712833116s) [upload=6063]: Got next segment. Inline: true Aug 7 22:22:12.125781404 (5h55m25.712837625s) [upload=6063]: Waiting for error group managing segments... Aug 7 22:22:12.125814315 (5h55m25.712870536s) [upload=6063]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:12.125818052 (5h55m25.712874263s) [upload=6063]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:12.125820397 (5h55m25.712876618s) [upload=6063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:12.024254277 (5h56m25.611310498s) [upload=6063]: Done uploading segments... Aug 7 22:23:12.024289072 (5h56m25.611345283s) [upload=6063]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:12.175471355 (5h56m25.762527616s) [upload=6082]: Starting upload Aug 7 22:23:12.175573175 (5h56m25.762629466s) [upload=6082]: Done starting upload Aug 7 22:23:12.175670356 (5h56m25.762726627s) [upload=6082]: Uploading segments... Aug 7 22:23:12.175760093 (5h56m25.762816364s) [upload=6082]: Got next segment. Inline: true Aug 7 22:23:12.175767677 (5h56m25.762823919s) [upload=6082]: Waiting for error group managing segments... Aug 7 22:23:12.175819345 (5h56m25.762875585s) [upload=6082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:12.175825105 (5h56m25.762881346s) [upload=6082]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:12.175828732 (5h56m25.762884973s) [upload=6082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:12.154559214 (5h57m25.741615455s) [upload=6082]: Done uploading segments... Aug 7 22:24:12.154578851 (5h57m25.741635092s) [upload=6082]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:12.253523032 (5h57m25.840579253s) [upload=6098]: Starting upload Aug 7 22:24:12.253572454 (5h57m25.840628665s) [upload=6098]: Done starting upload Aug 7 22:24:12.253615034 (5h57m25.840671245s) [upload=6098]: Uploading segments... Aug 7 22:24:12.253654808 (5h57m25.840711029s) [upload=6098]: Got next segment. Inline: true Aug 7 22:24:12.253658636 (5h57m25.840714847s) [upload=6098]: Waiting for error group managing segments... Aug 7 22:24:12.253689122 (5h57m25.840745343s) [upload=6098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.253692969 (5h57m25.84074918s) [upload=6098]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:12.253695303 (5h57m25.840751524s) [upload=6098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:12.467343237 (5h57m26.054399448s) [upload=6098]: Done uploading segments... Aug 7 22:24:12.467365428 (5h57m26.05442164s) [upload=6098]: Upload finished. err: <nil> Aug 7 22:24:33.085790933 (5h57m46.672847184s) [upload=6106]: Starting upload Aug 7 22:24:33.085870281 (5h57m46.672926522s) [upload=6106]: Done starting upload Aug 7 22:24:33.085932668 (5h57m46.67298892s) [upload=6106]: Uploading segments... Aug 7 22:24:33.085994053 (5h57m46.673050293s) [upload=6106]: Got next segment. Inline: true Aug 7 22:24:33.086000064 (5h57m46.673056304s) [upload=6106]: Waiting for error group managing segments... Aug 7 22:24:33.086045709 (5h57m46.67310195s) [upload=6106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:33.086051168 (5h57m46.67310741s) [upload=6106]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:33.086054985 (5h57m46.673111227s) [upload=6106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:33.072711385 (5h58m46.659767636s) [upload=6106]: Done uploading segments... Aug 7 22:25:33.072753384 (5h58m46.659809634s) [upload=6106]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:33.160012555 (5h58m46.747068795s) [upload=6120]: Starting upload Aug 7 22:25:33.160101981 (5h58m46.747158222s) [upload=6120]: Done starting upload Aug 7 22:25:33.160199473 (5h58m46.747255715s) [upload=6120]: Uploading segments... Aug 7 22:25:33.160243596 (5h58m46.747299837s) [upload=6120]: Got next segment. Inline: true Aug 7 22:25:33.160249046 (5h58m46.747305287s) [upload=6120]: Waiting for error group managing segments... Aug 7 22:25:33.160294231 (5h58m46.747350482s) [upload=6120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:33.160300012 (5h58m46.747356252s) [upload=6120]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:33.160303619 (5h58m46.747359859s) [upload=6120]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:33.228188865 (5h58m46.815245086s) [upload=6120]: Done uploading segments... Aug 7 22:25:33.228202571 (5h58m46.815258782s) [upload=6120]: Upload finished. err: <nil> Aug 7 22:26:14.972993755 (5h59m28.560049967s) [upload=6132]: Starting upload Aug 7 22:26:14.973047095 (5h59m28.560103307s) [upload=6132]: Done starting upload Aug 7 22:26:14.973101456 (5h59m28.560157677s) [upload=6132]: Uploading segments... Aug 7 22:26:14.973137704 (5h59m28.560193925s) [upload=6132]: Got next segment. Inline: true Aug 7 22:26:14.973141382 (5h59m28.560197592s) [upload=6132]: Waiting for error group managing segments... Aug 7 22:26:14.973187948 (5h59m28.560244158s) [upload=6132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:14.973191995 (5h59m28.560248217s) [upload=6132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:14.973195201 (5h59m28.560251423s) [upload=6132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:14.950324988 (6h0m28.537381229s) [upload=6132]: Done uploading segments... Aug 7 22:27:14.950345787 (6h0m28.537402028s) [upload=6132]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:22.925128081 (6h0m36.512184292s) [upload=6146]: Starting upload Aug 7 22:27:22.925194244 (6h0m36.512250465s) [upload=6146]: Done starting upload Aug 7 22:27:22.925235571 (6h0m36.512291783s) [upload=6146]: Uploading segments... Aug 7 22:27:22.925260508 (6h0m36.512316729s) [upload=6146]: Got next segment. Inline: true Aug 7 22:27:22.925264486 (6h0m36.512320707s) [upload=6146]: Waiting for error group managing segments... Aug 7 22:27:22.925305752 (6h0m36.512361973s) [upload=6146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:22.925309750 (6h0m36.512365971s) [upload=6146]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:22.925312044 (6h0m36.512368265s) [upload=6146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:22.913115749 (6h1m36.50017199s) [upload=6146]: Done uploading segments... Aug 7 22:28:22.913136929 (6h1m36.500193169s) [upload=6146]: Upload finished. err: metaclient: context deadline exceeded Duration: 215.976186ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 14.950859ms target: https://us1.storj.io/login FAIL (2025-08-07 22:51:44 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 14.980094ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 257.103275ms FAIL (2025-08-07 22:51:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 257.133502ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 12.493832ms target: https://us1.storj.io/login FAIL (2025-08-07 22:51:28 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 12.517425ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXX..X.............................] duration: 1.530580272s FAIL (2025-08-07 22:51:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.530601921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 22 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXX.XXXXX.X............................] duration: 205.159266ms FAIL (2025-08-07 22:51:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 420501s) [upload=6055]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:06.462569787 (5h55m20.049626008s) [upload=6061]: Starting upload Aug 7 22:22:06.462655577 (5h55m20.049711788s) [upload=6061]: Done starting upload Aug 7 22:22:06.462750936 (5h55m20.049807157s) [upload=6061]: Uploading segments... Aug 7 22:22:06.462782464 (5h55m20.049838675s) [upload=6061]: Got next segment. Inline: true Aug 7 22:22:06.462786181 (5h55m20.049842402s) [upload=6061]: Waiting for error group managing segments... Aug 7 22:22:06.462817791 (5h55m20.049874001s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:06.462821648 (5h55m20.049877869s) [upload=6061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:06.462823982 (5h55m20.049880203s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:36.449657257 (5h55m50.036713498s) [upload=6061]: Done uploading segments... Aug 7 22:22:36.449678196 (5h55m50.036734448s) [upload=6061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:59.911739779 (5h56m13.498795989s) [upload=6078]: Starting upload Aug 7 22:22:59.911786736 (5h56m13.498842957s) [upload=6078]: Done starting upload Aug 7 22:22:59.911827472 (5h56m13.498883683s) [upload=6078]: Uploading segments... Aug 7 22:22:59.911861074 (5h56m13.498917295s) [upload=6078]: Got next segment. Inline: true Aug 7 22:22:59.911864401 (5h56m13.498920622s) [upload=6078]: Waiting for error group managing segments... Aug 7 22:22:59.911889107 (5h56m13.498945319s) [upload=6078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:59.911893005 (5h56m13.498949226s) [upload=6078]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:59.911894898 (5h56m13.498951119s) [upload=6078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:29.900458747 (5h56m43.487514958s) [upload=6078]: Done uploading segments... Aug 7 22:23:29.900473835 (5h56m43.487530047s) [upload=6078]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:58.370463254 (5h57m11.957519465s) [upload=6094]: Starting upload Aug 7 22:23:58.370512907 (5h57m11.957569128s) [upload=6094]: Done starting upload Aug 7 22:23:58.370553031 (5h57m11.957609243s) [upload=6094]: Uploading segments... Aug 7 22:23:58.370578639 (5h57m11.95763486s) [upload=6094]: Got next segment. Inline: true Aug 7 22:23:58.370582427 (5h57m11.957638648s) [upload=6094]: Waiting for error group managing segments... Aug 7 22:23:58.370610951 (5h57m11.957667172s) [upload=6094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:58.370615178 (5h57m11.957671399s) [upload=6094]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:58.370617392 (5h57m11.957673613s) [upload=6094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:58.450972793 (5h57m12.038029004s) [upload=6094]: Done uploading segments... Aug 7 22:23:58.450986078 (5h57m12.038042289s) [upload=6094]: Upload finished. err: <nil> Aug 7 22:24:40.794473113 (5h57m54.381529325s) [upload=6107]: Starting upload Aug 7 22:24:40.794523428 (5h57m54.381579639s) [upload=6107]: Done starting upload Aug 7 22:24:40.794565206 (5h57m54.381621417s) [upload=6107]: Uploading segments... Aug 7 22:24:40.794591815 (5h57m54.381648026s) [upload=6107]: Got next segment. Inline: true Aug 7 22:24:40.794595622 (5h57m54.381651833s) [upload=6107]: Waiting for error group managing segments... Aug 7 22:24:40.794624367 (5h57m54.381680577s) [upload=6107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:40.794628323 (5h57m54.381684535s) [upload=6107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:40.794630638 (5h57m54.381686849s) [upload=6107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:10.774549311 (5h58m24.361605533s) [upload=6107]: Done uploading segments... Aug 7 22:25:10.774570019 (5h58m24.361626231s) [upload=6107]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:10.855666099 (5h58m24.442722339s) [upload=6114]: Starting upload Aug 7 22:25:10.855752680 (5h58m24.442808921s) [upload=6114]: Done starting upload Aug 7 22:25:10.855822740 (5h58m24.442878991s) [upload=6114]: Uploading segments... Aug 7 22:25:10.855866683 (5h58m24.442922923s) [upload=6114]: Got next segment. Inline: true Aug 7 22:25:10.855872564 (5h58m24.442928805s) [upload=6114]: Waiting for error group managing segments... Aug 7 22:25:10.855966008 (5h58m24.443022249s) [upload=6114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:10.855972971 (5h58m24.443029212s) [upload=6114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:10.855976568 (5h58m24.443032819s) [upload=6114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:40.842457902 (5h58m54.429514153s) [upload=6114]: Done uploading segments... Aug 7 22:25:40.842479021 (5h58m54.429535272s) [upload=6114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:40.940807525 (5h58m54.527863766s) [upload=6122]: Starting upload Aug 7 22:25:40.940896992 (5h58m54.527953233s) [upload=6122]: Done starting upload Aug 7 22:25:40.940964388 (5h58m54.528020628s) [upload=6122]: Uploading segments... Aug 7 22:25:40.941010434 (5h58m54.528066675s) [upload=6122]: Got next segment. Inline: true Aug 7 22:25:40.941016375 (5h58m54.528072616s) [upload=6122]: Waiting for error group managing segments... Aug 7 22:25:40.941060899 (5h58m54.52811715s) [upload=6122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:40.941068091 (5h58m54.528124332s) [upload=6122]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:40.941073060 (5h58m54.528129311s) [upload=6122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:10.929528068 (5h59m24.516584319s) [upload=6122]: Done uploading segments... Aug 7 22:26:10.929548557 (5h59m24.516604797s) [upload=6122]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:11.008231659 (5h59m24.59528788s) [upload=6129]: Starting upload Aug 7 22:26:11.008286501 (5h59m24.595342713s) [upload=6129]: Done starting upload Aug 7 22:26:11.008333198 (5h59m24.595389419s) [upload=6129]: Uploading segments... Aug 7 22:26:11.008364838 (5h59m24.595421059s) [upload=6129]: Got next segment. Inline: true Aug 7 22:26:11.008368595 (5h59m24.595424816s) [upload=6129]: Waiting for error group managing segments... Aug 7 22:26:11.008399943 (5h59m24.595456164s) [upload=6129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:11.008403599 (5h59m24.59545981s) [upload=6129]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:11.008406565 (5h59m24.595462786s) [upload=6129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:40.996305184 (5h59m54.583361385s) [upload=6129]: Done uploading segments... Aug 7 22:26:40.996323769 (5h59m54.58337998s) [upload=6129]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:14.829073707 (6h0m28.416129948s) [upload=6144]: Starting upload Aug 7 22:27:14.829153145 (6h0m28.416209386s) [upload=6144]: Done starting upload Aug 7 22:27:14.829247050 (6h0m28.416303291s) [upload=6144]: Uploading segments... Aug 7 22:27:14.829299899 (6h0m28.41635614s) [upload=6144]: Got next segment. Inline: true Aug 7 22:27:14.829306702 (6h0m28.416362943s) [upload=6144]: Waiting for error group managing segments... Aug 7 22:27:14.829350864 (6h0m28.416407105s) [upload=6144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:14.829357416 (6h0m28.416413657s) [upload=6144]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:14.829361143 (6h0m28.416417384s) [upload=6144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:44.808549878 (6h0m58.395606099s) [upload=6144]: Done uploading segments... Aug 7 22:27:44.808564877 (6h0m58.395621098s) [upload=6144]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:23.619901331 (6h1m37.206957573s) [upload=6162]: Starting upload Aug 7 22:28:23.620017008 (6h1m37.207073248s) [upload=6162]: Done starting upload Aug 7 22:28:23.620072551 (6h1m37.207128791s) [upload=6162]: Uploading segments... Aug 7 22:28:23.620125059 (6h1m37.2071813s) [upload=6162]: Got next segment. Inline: true Aug 7 22:28:23.620132432 (6h1m37.207188672s) [upload=6162]: Waiting for error group managing segments... Aug 7 22:28:23.620198335 (6h1m37.207254577s) [upload=6162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:23.620204407 (6h1m37.207260647s) [upload=6162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:23.620208153 (6h1m37.207264395s) [upload=6162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:23.683041871 (6h1m37.270098112s) [upload=6162]: Done uploading segments... Aug 7 22:28:23.683055537 (6h1m37.270111778s) [upload=6162]: Upload finished. err: <nil> Aug 7 22:28:40.804037302 (6h1m54.391093523s) [upload=6165]: Starting upload Aug 7 22:28:40.804089460 (6h1m54.391145671s) [upload=6165]: Done starting upload Aug 7 22:28:40.804134294 (6h1m54.391190515s) [upload=6165]: Uploading segments... Aug 7 22:28:40.804181422 (6h1m54.391237632s) [upload=6165]: Got next segment. Inline: true Aug 7 22:28:40.804185379 (6h1m54.3912416s) [upload=6165]: Waiting for error group managing segments... Aug 7 22:28:40.804217198 (6h1m54.391273419s) [upload=6165]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:40.804220915 (6h1m54.391277127s) [upload=6165]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:40.804223370 (6h1m54.391279591s) [upload=6165]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:10.793781791 (6h2m24.380838023s) [upload=6165]: Done uploading segments... Aug 7 22:29:10.793803392 (6h2m24.380859603s) [upload=6165]: Upload finished. err: metaclient: context deadline exceeded Duration: 205.20441ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXX.X.XXXX...............................] duration: 203.14334ms FAIL (2025-08-07 22:51:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 203.176752ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 14 out of 60 is failed (threshold=3) [..........XXXXXXXXXXX.X.XX..................................] duration: 162.502329ms FAIL (2025-08-07 22:51:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:04.377299713 (5h52m17.964355924s) [upload=6018]: Starting upload Aug 7 22:19:04.377350166 (5h52m17.964406387s) [upload=6018]: Done starting upload Aug 7 22:19:04.377401523 (5h52m17.964457744s) [upload=6018]: Uploading segments... Aug 7 22:19:04.377446897 (5h52m17.964503118s) [upload=6018]: Got next segment. Inline: true Aug 7 22:19:04.377453410 (5h52m17.96450962s) [upload=6018]: Waiting for error group managing segments... Aug 7 22:19:04.377487142 (5h52m17.964543354s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:04.377491361 (5h52m17.964547582s) [upload=6018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:04.377493875 (5h52m17.964550086s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:04.361964803 (5h53m17.949021024s) [upload=6018]: Done uploading segments... Aug 7 22:20:04.361979240 (5h53m17.949035461s) [upload=6018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:11.847632631 (5h53m25.434688843s) [upload=6035]: Starting upload Aug 7 22:20:11.847694537 (5h53m25.434750747s) [upload=6035]: Done starting upload Aug 7 22:20:11.847740192 (5h53m25.434796403s) [upload=6035]: Uploading segments... Aug 7 22:20:11.847772231 (5h53m25.434828442s) [upload=6035]: Got next segment. Inline: true Aug 7 22:20:11.847776860 (5h53m25.434833071s) [upload=6035]: Waiting for error group managing segments... Aug 7 22:20:11.847824679 (5h53m25.434880891s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:11.847830130 (5h53m25.434886341s) [upload=6035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:11.847833205 (5h53m25.434889417s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:11.836000640 (5h54m25.42305688s) [upload=6035]: Done uploading segments... Aug 7 22:21:11.836026087 (5h54m25.423082328s) [upload=6035]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:11.965120248 (5h54m25.552176489s) [upload=6050]: Starting upload Aug 7 22:21:11.965231515 (5h54m25.552287757s) [upload=6050]: Done starting upload Aug 7 22:21:11.965301156 (5h54m25.552357397s) [upload=6050]: Uploading segments... Aug 7 22:21:11.965348053 (5h54m25.552404304s) [upload=6050]: Got next segment. Inline: true Aug 7 22:21:11.965353994 (5h54m25.552410235s) [upload=6050]: Waiting for error group managing segments... Aug 7 22:21:11.965399049 (5h54m25.55245529s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:11.965405170 (5h54m25.552461411s) [upload=6050]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:11.965408957 (5h54m25.552465198s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:11.952333050 (5h55m25.539389261s) [upload=6050]: Done uploading segments... Aug 7 22:22:11.952348108 (5h55m25.539404329s) [upload=6050]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:12.125652873 (5h55m25.712709094s) [upload=6063]: Starting upload Aug 7 22:22:12.125702476 (5h55m25.712758688s) [upload=6063]: Done starting upload Aug 7 22:22:12.125747020 (5h55m25.712803241s) [upload=6063]: Uploading segments... Aug 7 22:22:12.125776895 (5h55m25.712833116s) [upload=6063]: Got next segment. Inline: true Aug 7 22:22:12.125781404 (5h55m25.712837625s) [upload=6063]: Waiting for error group managing segments... Aug 7 22:22:12.125814315 (5h55m25.712870536s) [upload=6063]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:12.125818052 (5h55m25.712874263s) [upload=6063]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:12.125820397 (5h55m25.712876618s) [upload=6063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:12.024254277 (5h56m25.611310498s) [upload=6063]: Done uploading segments... Aug 7 22:23:12.024289072 (5h56m25.611345283s) [upload=6063]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:12.175471355 (5h56m25.762527616s) [upload=6082]: Starting upload Aug 7 22:23:12.175573175 (5h56m25.762629466s) [upload=6082]: Done starting upload Aug 7 22:23:12.175670356 (5h56m25.762726627s) [upload=6082]: Uploading segments... Aug 7 22:23:12.175760093 (5h56m25.762816364s) [upload=6082]: Got next segment. Inline: true Aug 7 22:23:12.175767677 (5h56m25.762823919s) [upload=6082]: Waiting for error group managing segments... Aug 7 22:23:12.175819345 (5h56m25.762875585s) [upload=6082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:12.175825105 (5h56m25.762881346s) [upload=6082]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:12.175828732 (5h56m25.762884973s) [upload=6082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:12.154559214 (5h57m25.741615455s) [upload=6082]: Done uploading segments... Aug 7 22:24:12.154578851 (5h57m25.741635092s) [upload=6082]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:12.253523032 (5h57m25.840579253s) [upload=6098]: Starting upload Aug 7 22:24:12.253572454 (5h57m25.840628665s) [upload=6098]: Done starting upload Aug 7 22:24:12.253615034 (5h57m25.840671245s) [upload=6098]: Uploading segments... Aug 7 22:24:12.253654808 (5h57m25.840711029s) [upload=6098]: Got next segment. Inline: true Aug 7 22:24:12.253658636 (5h57m25.840714847s) [upload=6098]: Waiting for error group managing segments... Aug 7 22:24:12.253689122 (5h57m25.840745343s) [upload=6098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.253692969 (5h57m25.84074918s) [upload=6098]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:12.253695303 (5h57m25.840751524s) [upload=6098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:12.467343237 (5h57m26.054399448s) [upload=6098]: Done uploading segments... Aug 7 22:24:12.467365428 (5h57m26.05442164s) [upload=6098]: Upload finished. err: <nil> Aug 7 22:24:33.085790933 (5h57m46.672847184s) [upload=6106]: Starting upload Aug 7 22:24:33.085870281 (5h57m46.672926522s) [upload=6106]: Done starting upload Aug 7 22:24:33.085932668 (5h57m46.67298892s) [upload=6106]: Uploading segments... Aug 7 22:24:33.085994053 (5h57m46.673050293s) [upload=6106]: Got next segment. Inline: true Aug 7 22:24:33.086000064 (5h57m46.673056304s) [upload=6106]: Waiting for error group managing segments... Aug 7 22:24:33.086045709 (5h57m46.67310195s) [upload=6106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:33.086051168 (5h57m46.67310741s) [upload=6106]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:33.086054985 (5h57m46.673111227s) [upload=6106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:33.072711385 (5h58m46.659767636s) [upload=6106]: Done uploading segments... Aug 7 22:25:33.072753384 (5h58m46.659809634s) [upload=6106]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:33.160012555 (5h58m46.747068795s) [upload=6120]: Starting upload Aug 7 22:25:33.160101981 (5h58m46.747158222s) [upload=6120]: Done starting upload Aug 7 22:25:33.160199473 (5h58m46.747255715s) [upload=6120]: Uploading segments... Aug 7 22:25:33.160243596 (5h58m46.747299837s) [upload=6120]: Got next segment. Inline: true Aug 7 22:25:33.160249046 (5h58m46.747305287s) [upload=6120]: Waiting for error group managing segments... Aug 7 22:25:33.160294231 (5h58m46.747350482s) [upload=6120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:33.160300012 (5h58m46.747356252s) [upload=6120]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:33.160303619 (5h58m46.747359859s) [upload=6120]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:33.228188865 (5h58m46.815245086s) [upload=6120]: Done uploading segments... Aug 7 22:25:33.228202571 (5h58m46.815258782s) [upload=6120]: Upload finished. err: <nil> Aug 7 22:26:14.972993755 (5h59m28.560049967s) [upload=6132]: Starting upload Aug 7 22:26:14.973047095 (5h59m28.560103307s) [upload=6132]: Done starting upload Aug 7 22:26:14.973101456 (5h59m28.560157677s) [upload=6132]: Uploading segments... Aug 7 22:26:14.973137704 (5h59m28.560193925s) [upload=6132]: Got next segment. Inline: true Aug 7 22:26:14.973141382 (5h59m28.560197592s) [upload=6132]: Waiting for error group managing segments... Aug 7 22:26:14.973187948 (5h59m28.560244158s) [upload=6132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:14.973191995 (5h59m28.560248217s) [upload=6132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:14.973195201 (5h59m28.560251423s) [upload=6132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:14.950324988 (6h0m28.537381229s) [upload=6132]: Done uploading segments... Aug 7 22:27:14.950345787 (6h0m28.537402028s) [upload=6132]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:22.925128081 (6h0m36.512184292s) [upload=6146]: Starting upload Aug 7 22:27:22.925194244 (6h0m36.512250465s) [upload=6146]: Done starting upload Aug 7 22:27:22.925235571 (6h0m36.512291783s) [upload=6146]: Uploading segments... Aug 7 22:27:22.925260508 (6h0m36.512316729s) [upload=6146]: Got next segment. Inline: true Aug 7 22:27:22.925264486 (6h0m36.512320707s) [upload=6146]: Waiting for error group managing segments... Aug 7 22:27:22.925305752 (6h0m36.512361973s) [upload=6146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:22.925309750 (6h0m36.512365971s) [upload=6146]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:22.925312044 (6h0m36.512368265s) [upload=6146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:22.913115749 (6h1m36.50017199s) [upload=6146]: Done uploading segments... Aug 7 22:28:22.913136929 (6h1m36.500193169s) [upload=6146]: Upload finished. err: metaclient: context deadline exceeded Duration: 162.52413ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXX..XX..XX...............................] duration: 194.955516ms FAIL (2025-08-07 22:51:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 194.973118ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXX..X............................] duration: 1.645304528s FAIL (2025-08-07 22:51:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.645323924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....................XXXXXXXX.X.............................] duration: 1.593043457s FAIL (2025-08-07 22:51:05 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.59306664s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 6 out of 60 is failed (threshold=3) [.........................XX.XXXX............................] duration: 29.957789ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:51:04 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 29.974841ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 11.026132ms target: https://us1.storj.io/login FAIL (2025-08-07 22:51:04 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 11.120678ms core satellite us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 110.664789ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:51:03 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 110.689927ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 22 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXX.XXXXX.X...........................] duration: 181.802548ms FAIL (2025-08-07 22:51:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 420501s) [upload=6055]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:06.462569787 (5h55m20.049626008s) [upload=6061]: Starting upload Aug 7 22:22:06.462655577 (5h55m20.049711788s) [upload=6061]: Done starting upload Aug 7 22:22:06.462750936 (5h55m20.049807157s) [upload=6061]: Uploading segments... Aug 7 22:22:06.462782464 (5h55m20.049838675s) [upload=6061]: Got next segment. Inline: true Aug 7 22:22:06.462786181 (5h55m20.049842402s) [upload=6061]: Waiting for error group managing segments... Aug 7 22:22:06.462817791 (5h55m20.049874001s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:06.462821648 (5h55m20.049877869s) [upload=6061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:06.462823982 (5h55m20.049880203s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:36.449657257 (5h55m50.036713498s) [upload=6061]: Done uploading segments... Aug 7 22:22:36.449678196 (5h55m50.036734448s) [upload=6061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:59.911739779 (5h56m13.498795989s) [upload=6078]: Starting upload Aug 7 22:22:59.911786736 (5h56m13.498842957s) [upload=6078]: Done starting upload Aug 7 22:22:59.911827472 (5h56m13.498883683s) [upload=6078]: Uploading segments... Aug 7 22:22:59.911861074 (5h56m13.498917295s) [upload=6078]: Got next segment. Inline: true Aug 7 22:22:59.911864401 (5h56m13.498920622s) [upload=6078]: Waiting for error group managing segments... Aug 7 22:22:59.911889107 (5h56m13.498945319s) [upload=6078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:59.911893005 (5h56m13.498949226s) [upload=6078]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:59.911894898 (5h56m13.498951119s) [upload=6078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:29.900458747 (5h56m43.487514958s) [upload=6078]: Done uploading segments... Aug 7 22:23:29.900473835 (5h56m43.487530047s) [upload=6078]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:58.370463254 (5h57m11.957519465s) [upload=6094]: Starting upload Aug 7 22:23:58.370512907 (5h57m11.957569128s) [upload=6094]: Done starting upload Aug 7 22:23:58.370553031 (5h57m11.957609243s) [upload=6094]: Uploading segments... Aug 7 22:23:58.370578639 (5h57m11.95763486s) [upload=6094]: Got next segment. Inline: true Aug 7 22:23:58.370582427 (5h57m11.957638648s) [upload=6094]: Waiting for error group managing segments... Aug 7 22:23:58.370610951 (5h57m11.957667172s) [upload=6094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:58.370615178 (5h57m11.957671399s) [upload=6094]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:58.370617392 (5h57m11.957673613s) [upload=6094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:58.450972793 (5h57m12.038029004s) [upload=6094]: Done uploading segments... Aug 7 22:23:58.450986078 (5h57m12.038042289s) [upload=6094]: Upload finished. err: <nil> Aug 7 22:24:40.794473113 (5h57m54.381529325s) [upload=6107]: Starting upload Aug 7 22:24:40.794523428 (5h57m54.381579639s) [upload=6107]: Done starting upload Aug 7 22:24:40.794565206 (5h57m54.381621417s) [upload=6107]: Uploading segments... Aug 7 22:24:40.794591815 (5h57m54.381648026s) [upload=6107]: Got next segment. Inline: true Aug 7 22:24:40.794595622 (5h57m54.381651833s) [upload=6107]: Waiting for error group managing segments... Aug 7 22:24:40.794624367 (5h57m54.381680577s) [upload=6107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:40.794628323 (5h57m54.381684535s) [upload=6107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:40.794630638 (5h57m54.381686849s) [upload=6107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:10.774549311 (5h58m24.361605533s) [upload=6107]: Done uploading segments... Aug 7 22:25:10.774570019 (5h58m24.361626231s) [upload=6107]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:10.855666099 (5h58m24.442722339s) [upload=6114]: Starting upload Aug 7 22:25:10.855752680 (5h58m24.442808921s) [upload=6114]: Done starting upload Aug 7 22:25:10.855822740 (5h58m24.442878991s) [upload=6114]: Uploading segments... Aug 7 22:25:10.855866683 (5h58m24.442922923s) [upload=6114]: Got next segment. Inline: true Aug 7 22:25:10.855872564 (5h58m24.442928805s) [upload=6114]: Waiting for error group managing segments... Aug 7 22:25:10.855966008 (5h58m24.443022249s) [upload=6114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:10.855972971 (5h58m24.443029212s) [upload=6114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:10.855976568 (5h58m24.443032819s) [upload=6114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:40.842457902 (5h58m54.429514153s) [upload=6114]: Done uploading segments... Aug 7 22:25:40.842479021 (5h58m54.429535272s) [upload=6114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:40.940807525 (5h58m54.527863766s) [upload=6122]: Starting upload Aug 7 22:25:40.940896992 (5h58m54.527953233s) [upload=6122]: Done starting upload Aug 7 22:25:40.940964388 (5h58m54.528020628s) [upload=6122]: Uploading segments... Aug 7 22:25:40.941010434 (5h58m54.528066675s) [upload=6122]: Got next segment. Inline: true Aug 7 22:25:40.941016375 (5h58m54.528072616s) [upload=6122]: Waiting for error group managing segments... Aug 7 22:25:40.941060899 (5h58m54.52811715s) [upload=6122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:40.941068091 (5h58m54.528124332s) [upload=6122]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:40.941073060 (5h58m54.528129311s) [upload=6122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:10.929528068 (5h59m24.516584319s) [upload=6122]: Done uploading segments... Aug 7 22:26:10.929548557 (5h59m24.516604797s) [upload=6122]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:11.008231659 (5h59m24.59528788s) [upload=6129]: Starting upload Aug 7 22:26:11.008286501 (5h59m24.595342713s) [upload=6129]: Done starting upload Aug 7 22:26:11.008333198 (5h59m24.595389419s) [upload=6129]: Uploading segments... Aug 7 22:26:11.008364838 (5h59m24.595421059s) [upload=6129]: Got next segment. Inline: true Aug 7 22:26:11.008368595 (5h59m24.595424816s) [upload=6129]: Waiting for error group managing segments... Aug 7 22:26:11.008399943 (5h59m24.595456164s) [upload=6129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:11.008403599 (5h59m24.59545981s) [upload=6129]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:11.008406565 (5h59m24.595462786s) [upload=6129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:40.996305184 (5h59m54.583361385s) [upload=6129]: Done uploading segments... Aug 7 22:26:40.996323769 (5h59m54.58337998s) [upload=6129]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:14.829073707 (6h0m28.416129948s) [upload=6144]: Starting upload Aug 7 22:27:14.829153145 (6h0m28.416209386s) [upload=6144]: Done starting upload Aug 7 22:27:14.829247050 (6h0m28.416303291s) [upload=6144]: Uploading segments... Aug 7 22:27:14.829299899 (6h0m28.41635614s) [upload=6144]: Got next segment. Inline: true Aug 7 22:27:14.829306702 (6h0m28.416362943s) [upload=6144]: Waiting for error group managing segments... Aug 7 22:27:14.829350864 (6h0m28.416407105s) [upload=6144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:14.829357416 (6h0m28.416413657s) [upload=6144]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:14.829361143 (6h0m28.416417384s) [upload=6144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:44.808549878 (6h0m58.395606099s) [upload=6144]: Done uploading segments... Aug 7 22:27:44.808564877 (6h0m58.395621098s) [upload=6144]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:23.619901331 (6h1m37.206957573s) [upload=6162]: Starting upload Aug 7 22:28:23.620017008 (6h1m37.207073248s) [upload=6162]: Done starting upload Aug 7 22:28:23.620072551 (6h1m37.207128791s) [upload=6162]: Uploading segments... Aug 7 22:28:23.620125059 (6h1m37.2071813s) [upload=6162]: Got next segment. Inline: true Aug 7 22:28:23.620132432 (6h1m37.207188672s) [upload=6162]: Waiting for error group managing segments... Aug 7 22:28:23.620198335 (6h1m37.207254577s) [upload=6162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:23.620204407 (6h1m37.207260647s) [upload=6162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:23.620208153 (6h1m37.207264395s) [upload=6162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:23.683041871 (6h1m37.270098112s) [upload=6162]: Done uploading segments... Aug 7 22:28:23.683055537 (6h1m37.270111778s) [upload=6162]: Upload finished. err: <nil> Aug 7 22:28:40.804037302 (6h1m54.391093523s) [upload=6165]: Starting upload Aug 7 22:28:40.804089460 (6h1m54.391145671s) [upload=6165]: Done starting upload Aug 7 22:28:40.804134294 (6h1m54.391190515s) [upload=6165]: Uploading segments... Aug 7 22:28:40.804181422 (6h1m54.391237632s) [upload=6165]: Got next segment. Inline: true Aug 7 22:28:40.804185379 (6h1m54.3912416s) [upload=6165]: Waiting for error group managing segments... Aug 7 22:28:40.804217198 (6h1m54.391273419s) [upload=6165]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:40.804220915 (6h1m54.391277127s) [upload=6165]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:40.804223370 (6h1m54.391279591s) [upload=6165]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:10.793781791 (6h2m24.380838023s) [upload=6165]: Done uploading segments... Aug 7 22:29:10.793803392 (6h2m24.380859603s) [upload=6165]: Upload finished. err: metaclient: context deadline exceeded Duration: 181.82482ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXX..............................] duration: 948.826402ms FAIL (2025-08-07 22:50:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=6089]: Done starting upload Aug 7 22:23:43.961837605 (5h56m57.548893826s) [upload=6089]: Uploading segments... Aug 7 22:23:43.962138106 (5h56m57.549194347s) [upload=6089]: Got next segment. Inline: false Aug 7 22:23:43.962204581 (5h56m57.549260822s) [upload=6089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:43.962209980 (5h56m57.549266222s) [upload=6089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:12.375269699 (5h57m25.96232592s) [seg_pos={0 0}, upload=6089]: Begin upload segment... Aug 7 22:24:12.375313250 (5h57m25.962369471s) [seg_pos={0 0}, upload=6089]: Done begin upload segment. Aug 7 22:24:12.375339780 (5h57m25.962396031s) [noise=true , address=193.149.164.143:4698 , piece_id=AV2DCKYWWT5N4PL57WTXW45JNFYOCRLLEY6YSBSWOAQAPHAVXJQQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.375356050 (5h57m25.962412261s) [noise=true , address=104.36.82.53:4156 , piece_id=NBWZ4FB4S7CTDN2OTWTDKSH2R7DJN66P7GRTQTQ5FJZFQCZLDAQQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.375715481 (5h57m25.962771693s) [noise=true , address=104.36.82.58:4280 , piece_id=W2A65H4FIUOCUSMSVCANR2OPVBUFO4YNAFZV3KCASSCSPCHONGBA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.375724769 (5h57m25.96278099s) [noise=true , address=104.36.82.59:4333 , piece_id=JONL6XDWEUYMHVDPJVB54PE5HCYWN7KRO3FSNAMPJWCEP6QBKYVQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376022925 (5h57m25.963079146s) [noise=true , address=45.91.116.9:4270 , piece_id=CMJ7K7XDV44UUOWXU6NOUNMWZGLOEZKPTLVFMTNWIR6OK52TOEKQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376075263 (5h57m25.963131475s) [noise=true , address=104.36.82.36:4652 , piece_id=ZDJ2LSAX25MBAAW4PW745SX7J7X3S4PAGXEUS36T43KTM3LZ2WRQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376490167 (5h57m25.963546388s) [noise=true , address=104.36.82.45:4921 , piece_id=DCEBY7YTNAO4UR6YZ5LC5W27INFXZQ26BPJ7BHPMXY26P5BDL23Q, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376527327 (5h57m25.963583599s) [noise=true , address=104.36.82.12:4169 , piece_id=XPNXIWBPTKCEEQTVQ3BAQ3HCLQE5U5BTRIAR3FQ6O4OQJZMCJ6ZA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376755913 (5h57m25.963812125s) [noise=true , address=104.36.82.62:4411 , piece_id=3TMITFZMXF5ESOCJNLFU6EIT7CFUYULYZ6FJUQ6X7T2WBEQMKWFQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376864006 (5h57m25.963920216s) [noise=true , address=104.36.82.14:4243 , piece_id=WKT5D56ZXVSJJHVR3GB6N5XR6CGU4VW2XK2YTMPBZOH4S2ZPOVCA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376979641 (5h57m25.964035862s) [noise=true , address=104.36.82.9:4031 , piece_id=UOVVYSA77DQOK53ATDDXV56PLEVS6GBQ5HMZEKCEURYBPRCXNYPA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377222374 (5h57m25.964278595s) [noise=true , address=104.36.82.52:4100 , piece_id=7RUPILCVG3GL3KB4CS25YCYKDB4C6HFT6TGZWZIDC5EHBI2SD55Q, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377246499 (5h57m25.96430271s) [noise=true , address=79.127.201.215:4343 , piece_id=N3KGRRGXWJLCJGYSLPOLOIYZHP7AFWV6VBCHVBETDYOQBIXIWPMQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377620267 (5h57m25.964676488s) [noise=true , address=104.36.82.35:4597 , piece_id=6UMSPJQOVLEDWMIQMLAJLZ2AY6Y5XLMODCN3V63EB5JNJS3EC7BA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377642138 (5h57m25.964698349s) [noise=true , address=104.36.82.16:4300 , piece_id=RJRU7SETYXWMQK7GEE3HBD2XMEVSI3UJ6ZDCIWVW4266QSNAOPZA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377932059 (5h57m25.96498828s) [noise=true , address=104.36.82.47:4544 , piece_id=DVQRASYHRKPFEONZEWJ3MBVM7O5764NPWYADLMG4ARAH5PBD55MA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377950764 (5h57m25.965006985s) [noise=true , address=104.36.82.21:4478 , piece_id=5UO756WUOKCS52UJI2NIQFUSU7W6NBW34MGFH6YZX27HKWY5243A, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.378280479 (5h57m25.9653367s) [noise=true , address=104.36.82.37:4656 , piece_id=A25QOY4UFT7RRRY6PVAS3NNML7PGJM4OHDH56PF2HZHKJ7VOPURA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.468526485 (5h57m26.055582696s) [upload=6089]: Next returned nil segment Aug 7 22:24:12.468535081 (5h57m26.055591292s) [upload=6089]: Waiting for error group managing segments... Aug 7 22:24:12.483383738 (5h57m26.070439949s) [noise=true , address=104.36.82.37:4656 , piece_id=A25QOY4UFT7RRRY6PVAS3NNML7PGJM4OHDH56PF2HZHKJ7VOPURA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.500821799 (5h57m26.08787801s) [noise=true , address=104.36.82.16:4300 , piece_id=RJRU7SETYXWMQK7GEE3HBD2XMEVSI3UJ6ZDCIWVW4266QSNAOPZA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.504410257 (5h57m26.091466458s) [noise=true , address=45.91.116.9:4270 , piece_id=CMJ7K7XDV44UUOWXU6NOUNMWZGLOEZKPTLVFMTNWIR6OK52TOEKQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.511708921 (5h57m26.098765132s) [noise=true , address=104.36.82.35:4597 , piece_id=6UMSPJQOVLEDWMIQMLAJLZ2AY6Y5XLMODCN3V63EB5JNJS3EC7BA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.523758022 (5h57m26.110814233s) [noise=true , address=104.36.82.12:4169 , piece_id=XPNXIWBPTKCEEQTVQ3BAQ3HCLQE5U5BTRIAR3FQ6O4OQJZMCJ6ZA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.533614250 (5h57m26.120670461s) [noise=true , address=104.36.82.52:4100 , piece_id=7RUPILCVG3GL3KB4CS25YCYKDB4C6HFT6TGZWZIDC5EHBI2SD55Q, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.557032764 (5h57m26.144088985s) [noise=true , address=104.36.82.21:4478 , piece_id=5UO756WUOKCS52UJI2NIQFUSU7W6NBW34MGFH6YZX27HKWY5243A, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.582698011 (5h57m26.169754223s) [noise=true , address=104.36.82.53:4156 , piece_id=NBWZ4FB4S7CTDN2OTWTDKSH2R7DJN66P7GRTQTQ5FJZFQCZLDAQQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.609343038 (5h57m26.196399279s) [noise=true , address=79.127.201.215:4343 , piece_id=N3KGRRGXWJLCJGYSLPOLOIYZHP7AFWV6VBCHVBETDYOQBIXIWPMQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.639134328 (5h57m26.226190538s) [noise=true , address=104.36.82.36:4652 , piece_id=ZDJ2LSAX25MBAAW4PW745SX7J7X3S4PAGXEUS36T43KTM3LZ2WRQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.956879031 (5h57m26.543935272s) [noise=true , address=193.149.164.143:4698 , piece_id=AV2DCKYWWT5N4PL57WTXW45JNFYOCRLLEY6YSBSWOAQAPHAVXJQQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:13.036416979 (5h57m26.623473231s) [noise=true , address=104.36.82.9:4031 , piece_id=UOVVYSA77DQOK53ATDDXV56PLEVS6GBQ5HMZEKCEURYBPRCXNYPA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:13.172901316 (5h57m26.759957557s) [noise=true , address=104.36.82.47:4544 , piece_id=DVQRASYHRKPFEONZEWJ3MBVM7O5764NPWYADLMG4ARAH5PBD55MA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:13.202754351 (5h57m26.789810592s) [noise=true , address=104.36.82.14:4243 , piece_id=WKT5D56ZXVSJJHVR3GB6N5XR6CGU4VW2XK2YTMPBZOH4S2ZPOVCA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:13.202797742 (5h57m26.789853984s) [seg_pos={0 0}, upload=6089]: Segment reached optimal threshold of 14 pieces. Aug 7 22:24:13.203007333 (5h57m26.790063585s) [noise=true , address=104.36.82.58:4280 , piece_id=W2A65H4FIUOCUSMSVCANR2OPVBUFO4YNAFZV3KCASSCSPCHONGBA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fgHaEFKK4CeGg4x3vywTuYfT3kmac31xqg2QpsahQNXmvjhW8): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:13.203190164 (5h57m26.790246415s) [noise=true , address=104.36.82.59:4333 , piece_id=JONL6XDWEUYMHVDPJVB54PE5HCYWN7KRO3FSNAMPJWCEP6QBKYVQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K7U6NXcB6SFBtQFrG7DXt2skACr3u5DX2QsrnToi8EKywS9TN): protocol: write tcp 5.161.89.252:63730->104.36.82.59:4333: use of closed network connection; write tcp 5.161.89.252:63730->104.36.82.59:4333: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63730->104.36.82.59:4333: use of closed network connection Aug 7 22:24:13.203221153 (5h57m26.790277394s) [noise=true , address=104.36.82.45:4921 , piece_id=DCEBY7YTNAO4UR6YZ5LC5W27INFXZQ26BPJ7BHPMXY26P5BDL23Q, seg_pos={0 0}, upload=6089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bBRKXiuSN2cw6KrdP8q4Dsf4MiFCUSgMEGjqG2dKL9RLTQra1): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:13.203424783 (5h57m26.790481023s) [noise=true , address=104.36.82.62:4411 , piece_id=3TMITFZMXF5ESOCJNLFU6EIT7CFUYULYZ6FJUQ6X7T2WBEQMKWFQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: context canceled; context canceled Aug 7 22:24:13.203452675 (5h57m26.790508916s) [seg_pos={0 0}, upload=6089]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:24:13.203591695 (5h57m26.790647935s) [upload=6089]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:13.203602313 (5h57m26.790658565s) [upload=6089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:03.852878645 (6h2m17.439934866s) [upload=6089]: Done uploading segments... Aug 7 22:29:03.852904053 (6h2m17.439960294s) [upload=6089]: Upload finished. err: metaclient: context deadline exceeded Duration: 948.844947ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXX..XXXX.X............................] duration: 564.255073ms FAIL (2025-08-07 22:50:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 564.492275ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 14 out of 60 is failed (threshold=3) [...........XXXXXXXXXXX.X.XX.................................] duration: 197.0893ms FAIL (2025-08-07 22:50:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:04.377299713 (5h52m17.964355924s) [upload=6018]: Starting upload Aug 7 22:19:04.377350166 (5h52m17.964406387s) [upload=6018]: Done starting upload Aug 7 22:19:04.377401523 (5h52m17.964457744s) [upload=6018]: Uploading segments... Aug 7 22:19:04.377446897 (5h52m17.964503118s) [upload=6018]: Got next segment. Inline: true Aug 7 22:19:04.377453410 (5h52m17.96450962s) [upload=6018]: Waiting for error group managing segments... Aug 7 22:19:04.377487142 (5h52m17.964543354s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:04.377491361 (5h52m17.964547582s) [upload=6018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:04.377493875 (5h52m17.964550086s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:04.361964803 (5h53m17.949021024s) [upload=6018]: Done uploading segments... Aug 7 22:20:04.361979240 (5h53m17.949035461s) [upload=6018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:11.847632631 (5h53m25.434688843s) [upload=6035]: Starting upload Aug 7 22:20:11.847694537 (5h53m25.434750747s) [upload=6035]: Done starting upload Aug 7 22:20:11.847740192 (5h53m25.434796403s) [upload=6035]: Uploading segments... Aug 7 22:20:11.847772231 (5h53m25.434828442s) [upload=6035]: Got next segment. Inline: true Aug 7 22:20:11.847776860 (5h53m25.434833071s) [upload=6035]: Waiting for error group managing segments... Aug 7 22:20:11.847824679 (5h53m25.434880891s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:11.847830130 (5h53m25.434886341s) [upload=6035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:11.847833205 (5h53m25.434889417s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:11.836000640 (5h54m25.42305688s) [upload=6035]: Done uploading segments... Aug 7 22:21:11.836026087 (5h54m25.423082328s) [upload=6035]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:11.965120248 (5h54m25.552176489s) [upload=6050]: Starting upload Aug 7 22:21:11.965231515 (5h54m25.552287757s) [upload=6050]: Done starting upload Aug 7 22:21:11.965301156 (5h54m25.552357397s) [upload=6050]: Uploading segments... Aug 7 22:21:11.965348053 (5h54m25.552404304s) [upload=6050]: Got next segment. Inline: true Aug 7 22:21:11.965353994 (5h54m25.552410235s) [upload=6050]: Waiting for error group managing segments... Aug 7 22:21:11.965399049 (5h54m25.55245529s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:11.965405170 (5h54m25.552461411s) [upload=6050]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:11.965408957 (5h54m25.552465198s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:11.952333050 (5h55m25.539389261s) [upload=6050]: Done uploading segments... Aug 7 22:22:11.952348108 (5h55m25.539404329s) [upload=6050]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:12.125652873 (5h55m25.712709094s) [upload=6063]: Starting upload Aug 7 22:22:12.125702476 (5h55m25.712758688s) [upload=6063]: Done starting upload Aug 7 22:22:12.125747020 (5h55m25.712803241s) [upload=6063]: Uploading segments... Aug 7 22:22:12.125776895 (5h55m25.712833116s) [upload=6063]: Got next segment. Inline: true Aug 7 22:22:12.125781404 (5h55m25.712837625s) [upload=6063]: Waiting for error group managing segments... Aug 7 22:22:12.125814315 (5h55m25.712870536s) [upload=6063]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:12.125818052 (5h55m25.712874263s) [upload=6063]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:12.125820397 (5h55m25.712876618s) [upload=6063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:12.024254277 (5h56m25.611310498s) [upload=6063]: Done uploading segments... Aug 7 22:23:12.024289072 (5h56m25.611345283s) [upload=6063]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:12.175471355 (5h56m25.762527616s) [upload=6082]: Starting upload Aug 7 22:23:12.175573175 (5h56m25.762629466s) [upload=6082]: Done starting upload Aug 7 22:23:12.175670356 (5h56m25.762726627s) [upload=6082]: Uploading segments... Aug 7 22:23:12.175760093 (5h56m25.762816364s) [upload=6082]: Got next segment. Inline: true Aug 7 22:23:12.175767677 (5h56m25.762823919s) [upload=6082]: Waiting for error group managing segments... Aug 7 22:23:12.175819345 (5h56m25.762875585s) [upload=6082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:12.175825105 (5h56m25.762881346s) [upload=6082]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:12.175828732 (5h56m25.762884973s) [upload=6082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:12.154559214 (5h57m25.741615455s) [upload=6082]: Done uploading segments... Aug 7 22:24:12.154578851 (5h57m25.741635092s) [upload=6082]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:12.253523032 (5h57m25.840579253s) [upload=6098]: Starting upload Aug 7 22:24:12.253572454 (5h57m25.840628665s) [upload=6098]: Done starting upload Aug 7 22:24:12.253615034 (5h57m25.840671245s) [upload=6098]: Uploading segments... Aug 7 22:24:12.253654808 (5h57m25.840711029s) [upload=6098]: Got next segment. Inline: true Aug 7 22:24:12.253658636 (5h57m25.840714847s) [upload=6098]: Waiting for error group managing segments... Aug 7 22:24:12.253689122 (5h57m25.840745343s) [upload=6098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.253692969 (5h57m25.84074918s) [upload=6098]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:12.253695303 (5h57m25.840751524s) [upload=6098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:12.467343237 (5h57m26.054399448s) [upload=6098]: Done uploading segments... Aug 7 22:24:12.467365428 (5h57m26.05442164s) [upload=6098]: Upload finished. err: <nil> Aug 7 22:24:33.085790933 (5h57m46.672847184s) [upload=6106]: Starting upload Aug 7 22:24:33.085870281 (5h57m46.672926522s) [upload=6106]: Done starting upload Aug 7 22:24:33.085932668 (5h57m46.67298892s) [upload=6106]: Uploading segments... Aug 7 22:24:33.085994053 (5h57m46.673050293s) [upload=6106]: Got next segment. Inline: true Aug 7 22:24:33.086000064 (5h57m46.673056304s) [upload=6106]: Waiting for error group managing segments... Aug 7 22:24:33.086045709 (5h57m46.67310195s) [upload=6106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:33.086051168 (5h57m46.67310741s) [upload=6106]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:33.086054985 (5h57m46.673111227s) [upload=6106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:33.072711385 (5h58m46.659767636s) [upload=6106]: Done uploading segments... Aug 7 22:25:33.072753384 (5h58m46.659809634s) [upload=6106]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:33.160012555 (5h58m46.747068795s) [upload=6120]: Starting upload Aug 7 22:25:33.160101981 (5h58m46.747158222s) [upload=6120]: Done starting upload Aug 7 22:25:33.160199473 (5h58m46.747255715s) [upload=6120]: Uploading segments... Aug 7 22:25:33.160243596 (5h58m46.747299837s) [upload=6120]: Got next segment. Inline: true Aug 7 22:25:33.160249046 (5h58m46.747305287s) [upload=6120]: Waiting for error group managing segments... Aug 7 22:25:33.160294231 (5h58m46.747350482s) [upload=6120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:33.160300012 (5h58m46.747356252s) [upload=6120]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:33.160303619 (5h58m46.747359859s) [upload=6120]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:33.228188865 (5h58m46.815245086s) [upload=6120]: Done uploading segments... Aug 7 22:25:33.228202571 (5h58m46.815258782s) [upload=6120]: Upload finished. err: <nil> Aug 7 22:26:14.972993755 (5h59m28.560049967s) [upload=6132]: Starting upload Aug 7 22:26:14.973047095 (5h59m28.560103307s) [upload=6132]: Done starting upload Aug 7 22:26:14.973101456 (5h59m28.560157677s) [upload=6132]: Uploading segments... Aug 7 22:26:14.973137704 (5h59m28.560193925s) [upload=6132]: Got next segment. Inline: true Aug 7 22:26:14.973141382 (5h59m28.560197592s) [upload=6132]: Waiting for error group managing segments... Aug 7 22:26:14.973187948 (5h59m28.560244158s) [upload=6132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:14.973191995 (5h59m28.560248217s) [upload=6132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:14.973195201 (5h59m28.560251423s) [upload=6132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:14.950324988 (6h0m28.537381229s) [upload=6132]: Done uploading segments... Aug 7 22:27:14.950345787 (6h0m28.537402028s) [upload=6132]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:22.925128081 (6h0m36.512184292s) [upload=6146]: Starting upload Aug 7 22:27:22.925194244 (6h0m36.512250465s) [upload=6146]: Done starting upload Aug 7 22:27:22.925235571 (6h0m36.512291783s) [upload=6146]: Uploading segments... Aug 7 22:27:22.925260508 (6h0m36.512316729s) [upload=6146]: Got next segment. Inline: true Aug 7 22:27:22.925264486 (6h0m36.512320707s) [upload=6146]: Waiting for error group managing segments... Aug 7 22:27:22.925305752 (6h0m36.512361973s) [upload=6146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:22.925309750 (6h0m36.512365971s) [upload=6146]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:22.925312044 (6h0m36.512368265s) [upload=6146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:22.913115749 (6h1m36.50017199s) [upload=6146]: Done uploading segments... Aug 7 22:28:22.913136929 (6h1m36.500193169s) [upload=6146]: Upload finished. err: metaclient: context deadline exceeded Duration: 197.104479ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXX.X.XXXX..............................] duration: 205.919424ms FAIL (2025-08-07 22:50:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 205.939401ms core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 16.389373ms target: https://us1.storj.io/login FAIL (2025-08-07 22:50:47 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 16.414839ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 198.268291ms FAIL (2025-08-07 22:50:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 198.292236ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXX.XXX.................................] duration: 471.977585ms FAIL (2025-08-07 22:50:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== TVRWLFHTOF2YZFDYX3LU5FENJAS6JOW572GLJ2OHVQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.749054955 (5h56m13.336111176s) [noise=true , address=104.36.82.2:4730 , piece_id=FKFZNBUC7L2WVTTFPNRSPJUWW6HAEE6CNUAYSHVDYXW5XENS462Q, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.749426248 (5h56m13.336482459s) [noise=true , address=45.91.116.9:4262 , piece_id=R7J2NVZJTXR7ATH2VUC75B2Z73TSJAQUBYJ3WE75RQ6AD3L3PSCQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.750681361 (5h56m13.337737582s) [noise=true , address=104.36.82.5:4840 , piece_id=6SYENOMYMIEYCEFSHJKMXOQE4RCDPDWBAHYDVRTXCQYWJGQDCCKA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.751093369 (5h56m13.338149581s) [noise=true , address=104.36.82.14:4221 , piece_id=SBKEVRBOXXBFOVTZWPSDVDPXEDV6HZIVAV4MWFIYDMNEU643KBWA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.751643326 (5h56m13.338699547s) [noise=true , address=45.91.116.4:4940 , piece_id=UXKWVJQC6KYUHGXKHRYHJATOP5KQF4BY2LWNQJQEGFYKQEBT6GWA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.752249137 (5h56m13.339305348s) [noise=true , address=77.90.31.13:4628 , piece_id=BYQWTXNUD76XIWZZAUZNHDUX7F7CABMDCPK7M7QPDHS3CFBRBMVQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.752826465 (5h56m13.339882676s) [noise=true , address=104.36.82.2:4731 , piece_id=L6W56WDLHKGHEPXBK7F2OYZ6CRM66ZVKG5RGJPH3FHC7RZZ3QGBA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.753473372 (5h56m13.340529583s) [noise=true , address=45.91.116.12:4227 , piece_id=7RIB2VCODFSHTRTBGI3Z74IECO3ATQXSXW7CY3TCA43MZKSKEQYA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.753961884 (5h56m13.341018096s) [noise=true , address=77.90.31.10:4519 , piece_id=QF2SVPCWB5HG4S7GQL6NFHTYPVOHP4HHORRY2TMJSRXP5A3J43TQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.754616727 (5h56m13.341672948s) [noise=true , address=104.36.82.9:4006 , piece_id=MDXFW3M6TBQJXYHMBTPIOD5I5PK3NUHDRZ3EERNCRKFVMUPSW43Q, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.755122451 (5h56m13.342178672s) [noise=true , address=104.36.82.8:4971 , piece_id=2MZZBGEPYJSHOBAG4FUAU5NKKAOL72C2AWD3GZECQXGPMOG6TDMQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.837536198 (5h56m13.424592419s) [upload=6077]: Next returned nil segment Aug 7 22:22:59.837547418 (5h56m13.424603629s) [upload=6077]: Waiting for error group managing segments... Aug 7 22:22:59.846810959 (5h56m13.43386718s) [noise=true , address=45.91.116.7:4011 , piece_id=E7YGVXQXIO2A34B22AHGKBQXER4KLEX3AFGHVFUQ6USLQMUFSW2Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.847279324 (5h56m13.434335535s) [noise=true , address=45.91.116.12:4227 , piece_id=7RIB2VCODFSHTRTBGI3Z74IECO3ATQXSXW7CY3TCA43MZKSKEQYA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.848559454 (5h56m13.435615675s) [noise=true , address=45.91.116.4:4940 , piece_id=UXKWVJQC6KYUHGXKHRYHJATOP5KQF4BY2LWNQJQEGFYKQEBT6GWA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.852382881 (5h56m13.439439102s) [noise=true , address=45.91.116.9:4262 , piece_id=R7J2NVZJTXR7ATH2VUC75B2Z73TSJAQUBYJ3WE75RQ6AD3L3PSCQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.852577974 (5h56m13.439634195s) [noise=true , address=77.90.31.19:4842 , piece_id=ULCHXICRO3ULX7IEW6GVXHHGXT6F42N3WQDEEUVOSKCXID6F6XZQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.856642091 (5h56m13.443698302s) [noise=true , address=104.36.82.8:4971 , piece_id=2MZZBGEPYJSHOBAG4FUAU5NKKAOL72C2AWD3GZECQXGPMOG6TDMQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.856766534 (5h56m13.443822745s) [noise=true , address=77.90.31.10:4519 , piece_id=QF2SVPCWB5HG4S7GQL6NFHTYPVOHP4HHORRY2TMJSRXP5A3J43TQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.868085922 (5h56m13.455142143s) [noise=true , address=45.91.116.12:4225 , piece_id=MNJGBICBIG3HQXCO7B2C2ITRFSOW475RAGNVTAQNVFQYAJ7DOU5Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.868246762 (5h56m13.455302973s) [noise=true , address=104.36.82.2:4730 , piece_id=FKFZNBUC7L2WVTTFPNRSPJUWW6HAEE6CNUAYSHVDYXW5XENS462Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.877804453 (5h56m13.464860674s) [noise=true , address=104.36.82.11:4115 , piece_id=BALIRX6S5WW2CGSXSIFHWV2ABDH3ASPHR36YEYRUYTTGZCW6U6AQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.882357252 (5h56m13.469413462s) [noise=true , address=104.36.82.2:4731 , piece_id=L6W56WDLHKGHEPXBK7F2OYZ6CRM66ZVKG5RGJPH3FHC7RZZ3QGBA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.887398722 (5h56m13.474454933s) [noise=true , address=104.36.82.14:4221 , piece_id=SBKEVRBOXXBFOVTZWPSDVDPXEDV6HZIVAV4MWFIYDMNEU643KBWA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.887674137 (5h56m13.474730358s) [noise=true , address=104.36.82.10:4079 , piece_id=DHLJEMGMQUTVRWLFHTOF2YZFDYX3LU5FENJAS6JOW572GLJ2OHVQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.887689665 (5h56m13.474745877s) [seg_pos={0 0}, upload=6077]: Segment reached optimal threshold of 13 pieces. Aug 7 22:22:59.887775385 (5h56m13.474831606s) [noise=true , address=77.90.31.13:4628 , piece_id=BYQWTXNUD76XIWZZAUZNHDUX7F7CABMDCPK7M7QPDHS3CFBRBMVQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Aug 7 22:22:59.887795453 (5h56m13.474851674s) [noise=true , address=104.36.82.5:4840 , piece_id=6SYENOMYMIEYCEFSHJKMXOQE4RCDPDWBAHYDVRTXCQYWJGQDCCKA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Aug 7 22:22:59.887879469 (5h56m13.47493569s) [noise=true , address=104.36.82.9:4006 , piece_id=MDXFW3M6TBQJXYHMBTPIOD5I5PK3NUHDRZ3EERNCRKFVMUPSW43Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 5.161.89.252:59712->104.36.82.9:4006: use of closed network connection; write tcp 5.161.89.252:59712->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:59712->104.36.82.9:4006: use of closed network connection Aug 7 22:22:59.887943669 (5h56m13.47499988s) [seg_pos={0 0}, upload=6077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:22:59.888006888 (5h56m13.475063109s) [upload=6077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:22:59.888010665 (5h56m13.475066886s) [upload=6077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:59.933305033 (5h56m13.520361243s) [upload=6077]: Done uploading segments... Aug 7 22:22:59.933317746 (5h56m13.520373958s) [upload=6077]: Upload finished. err: <nil> Aug 7 22:23:47.739190890 (5h57m1.326247131s) [upload=6091]: Starting upload Aug 7 22:23:47.739297178 (5h57m1.326353419s) [upload=6091]: Done starting upload Aug 7 22:23:47.739428053 (5h57m1.326484274s) [upload=6091]: Uploading segments... Aug 7 22:23:47.739777785 (5h57m1.326834025s) [upload=6091]: Got next segment. Inline: false Aug 7 22:23:47.739805196 (5h57m1.326861407s) [upload=6091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:47.739811638 (5h57m1.326867849s) [upload=6091]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:17.722308743 (5h57m31.309364985s) [upload=6091]: Next segment err: context deadline exceeded Aug 7 22:24:17.722438996 (5h57m31.309495238s) [upload=6091]: Done uploading segments... Aug 7 22:24:17.722476005 (5h57m31.309532246s) [upload=6091]: Upload finished. err: context deadline exceeded Aug 7 22:24:18.676382471 (5h57m32.263438711s) [upload=6101]: Starting upload Aug 7 22:24:18.676482739 (5h57m32.263538979s) [upload=6101]: Done starting upload Aug 7 22:24:18.676619723 (5h57m32.263675964s) [upload=6101]: Uploading segments... Aug 7 22:24:18.676886642 (5h57m32.263942883s) [upload=6101]: Got next segment. Inline: false Aug 7 22:24:18.676922940 (5h57m32.263979181s) [upload=6101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:18.676930835 (5h57m32.263987076s) [upload=6101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:48.662510024 (5h58m2.249566255s) [upload=6101]: Next segment err: context deadline exceeded Aug 7 22:24:48.662708235 (5h58m2.249764465s) [upload=6101]: Done uploading segments... Aug 7 22:24:48.662735284 (5h58m2.249791526s) [upload=6101]: Upload finished. err: context deadline exceeded Aug 7 22:25:18.142081018 (5h58m31.729137259s) [upload=6117]: Starting upload Aug 7 22:25:18.142160586 (5h58m31.729216828s) [upload=6117]: Done starting upload Aug 7 22:25:18.142286381 (5h58m31.729342623s) [upload=6117]: Uploading segments... Aug 7 22:25:18.142594587 (5h58m31.729650828s) [upload=6117]: Got next segment. Inline: false Aug 7 22:25:18.142641995 (5h58m31.729698236s) [upload=6117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:18.142647436 (5h58m31.729703676s) [upload=6117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:25:48.125202589 (5h59m1.71225884s) [upload=6117]: Next segment err: context deadline exceeded Aug 7 22:25:48.125308446 (5h59m1.712364688s) [upload=6117]: Done uploading segments... Aug 7 22:25:48.125355273 (5h59m1.712411525s) [upload=6117]: Upload finished. err: context deadline exceeded Duration: 472.005868ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXX..XX..XX..............................] duration: 193.291782ms FAIL (2025-08-07 22:50:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 193.310097ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXX..XXXX.X...........................] duration: 396.530315ms FAIL (2025-08-07 22:50:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 396.550833ms edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 12.529988ms target: https://us1.storj.io/login FAIL (2025-08-07 22:50:31 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 12.550587ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXX.............................] duration: 997.828171ms FAIL (2025-08-07 22:50:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=6089]: Done starting upload Aug 7 22:23:43.961837605 (5h56m57.548893826s) [upload=6089]: Uploading segments... Aug 7 22:23:43.962138106 (5h56m57.549194347s) [upload=6089]: Got next segment. Inline: false Aug 7 22:23:43.962204581 (5h56m57.549260822s) [upload=6089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:43.962209980 (5h56m57.549266222s) [upload=6089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:12.375269699 (5h57m25.96232592s) [seg_pos={0 0}, upload=6089]: Begin upload segment... Aug 7 22:24:12.375313250 (5h57m25.962369471s) [seg_pos={0 0}, upload=6089]: Done begin upload segment. Aug 7 22:24:12.375339780 (5h57m25.962396031s) [noise=true , address=193.149.164.143:4698 , piece_id=AV2DCKYWWT5N4PL57WTXW45JNFYOCRLLEY6YSBSWOAQAPHAVXJQQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.375356050 (5h57m25.962412261s) [noise=true , address=104.36.82.53:4156 , piece_id=NBWZ4FB4S7CTDN2OTWTDKSH2R7DJN66P7GRTQTQ5FJZFQCZLDAQQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.375715481 (5h57m25.962771693s) [noise=true , address=104.36.82.58:4280 , piece_id=W2A65H4FIUOCUSMSVCANR2OPVBUFO4YNAFZV3KCASSCSPCHONGBA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.375724769 (5h57m25.96278099s) [noise=true , address=104.36.82.59:4333 , piece_id=JONL6XDWEUYMHVDPJVB54PE5HCYWN7KRO3FSNAMPJWCEP6QBKYVQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376022925 (5h57m25.963079146s) [noise=true , address=45.91.116.9:4270 , piece_id=CMJ7K7XDV44UUOWXU6NOUNMWZGLOEZKPTLVFMTNWIR6OK52TOEKQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376075263 (5h57m25.963131475s) [noise=true , address=104.36.82.36:4652 , piece_id=ZDJ2LSAX25MBAAW4PW745SX7J7X3S4PAGXEUS36T43KTM3LZ2WRQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376490167 (5h57m25.963546388s) [noise=true , address=104.36.82.45:4921 , piece_id=DCEBY7YTNAO4UR6YZ5LC5W27INFXZQ26BPJ7BHPMXY26P5BDL23Q, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376527327 (5h57m25.963583599s) [noise=true , address=104.36.82.12:4169 , piece_id=XPNXIWBPTKCEEQTVQ3BAQ3HCLQE5U5BTRIAR3FQ6O4OQJZMCJ6ZA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376755913 (5h57m25.963812125s) [noise=true , address=104.36.82.62:4411 , piece_id=3TMITFZMXF5ESOCJNLFU6EIT7CFUYULYZ6FJUQ6X7T2WBEQMKWFQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376864006 (5h57m25.963920216s) [noise=true , address=104.36.82.14:4243 , piece_id=WKT5D56ZXVSJJHVR3GB6N5XR6CGU4VW2XK2YTMPBZOH4S2ZPOVCA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.376979641 (5h57m25.964035862s) [noise=true , address=104.36.82.9:4031 , piece_id=UOVVYSA77DQOK53ATDDXV56PLEVS6GBQ5HMZEKCEURYBPRCXNYPA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377222374 (5h57m25.964278595s) [noise=true , address=104.36.82.52:4100 , piece_id=7RUPILCVG3GL3KB4CS25YCYKDB4C6HFT6TGZWZIDC5EHBI2SD55Q, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377246499 (5h57m25.96430271s) [noise=true , address=79.127.201.215:4343 , piece_id=N3KGRRGXWJLCJGYSLPOLOIYZHP7AFWV6VBCHVBETDYOQBIXIWPMQ, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377620267 (5h57m25.964676488s) [noise=true , address=104.36.82.35:4597 , piece_id=6UMSPJQOVLEDWMIQMLAJLZ2AY6Y5XLMODCN3V63EB5JNJS3EC7BA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377642138 (5h57m25.964698349s) [noise=true , address=104.36.82.16:4300 , piece_id=RJRU7SETYXWMQK7GEE3HBD2XMEVSI3UJ6ZDCIWVW4266QSNAOPZA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377932059 (5h57m25.96498828s) [noise=true , address=104.36.82.47:4544 , piece_id=DVQRASYHRKPFEONZEWJ3MBVM7O5764NPWYADLMG4ARAH5PBD55MA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.377950764 (5h57m25.965006985s) [noise=true , address=104.36.82.21:4478 , piece_id=5UO756WUOKCS52UJI2NIQFUSU7W6NBW34MGFH6YZX27HKWY5243A, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.378280479 (5h57m25.9653367s) [noise=true , address=104.36.82.37:4656 , piece_id=A25QOY4UFT7RRRY6PVAS3NNML7PGJM4OHDH56PF2HZHKJ7VOPURA, seg_pos={0 0}, upload=6089]: Uploading piece... Aug 7 22:24:12.468526485 (5h57m26.055582696s) [upload=6089]: Next returned nil segment Aug 7 22:24:12.468535081 (5h57m26.055591292s) [upload=6089]: Waiting for error group managing segments... Aug 7 22:24:12.483383738 (5h57m26.070439949s) [noise=true , address=104.36.82.37:4656 , piece_id=A25QOY4UFT7RRRY6PVAS3NNML7PGJM4OHDH56PF2HZHKJ7VOPURA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.500821799 (5h57m26.08787801s) [noise=true , address=104.36.82.16:4300 , piece_id=RJRU7SETYXWMQK7GEE3HBD2XMEVSI3UJ6ZDCIWVW4266QSNAOPZA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.504410257 (5h57m26.091466458s) [noise=true , address=45.91.116.9:4270 , piece_id=CMJ7K7XDV44UUOWXU6NOUNMWZGLOEZKPTLVFMTNWIR6OK52TOEKQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.511708921 (5h57m26.098765132s) [noise=true , address=104.36.82.35:4597 , piece_id=6UMSPJQOVLEDWMIQMLAJLZ2AY6Y5XLMODCN3V63EB5JNJS3EC7BA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.523758022 (5h57m26.110814233s) [noise=true , address=104.36.82.12:4169 , piece_id=XPNXIWBPTKCEEQTVQ3BAQ3HCLQE5U5BTRIAR3FQ6O4OQJZMCJ6ZA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.533614250 (5h57m26.120670461s) [noise=true , address=104.36.82.52:4100 , piece_id=7RUPILCVG3GL3KB4CS25YCYKDB4C6HFT6TGZWZIDC5EHBI2SD55Q, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.557032764 (5h57m26.144088985s) [noise=true , address=104.36.82.21:4478 , piece_id=5UO756WUOKCS52UJI2NIQFUSU7W6NBW34MGFH6YZX27HKWY5243A, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.582698011 (5h57m26.169754223s) [noise=true , address=104.36.82.53:4156 , piece_id=NBWZ4FB4S7CTDN2OTWTDKSH2R7DJN66P7GRTQTQ5FJZFQCZLDAQQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.609343038 (5h57m26.196399279s) [noise=true , address=79.127.201.215:4343 , piece_id=N3KGRRGXWJLCJGYSLPOLOIYZHP7AFWV6VBCHVBETDYOQBIXIWPMQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.639134328 (5h57m26.226190538s) [noise=true , address=104.36.82.36:4652 , piece_id=ZDJ2LSAX25MBAAW4PW745SX7J7X3S4PAGXEUS36T43KTM3LZ2WRQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:12.956879031 (5h57m26.543935272s) [noise=true , address=193.149.164.143:4698 , piece_id=AV2DCKYWWT5N4PL57WTXW45JNFYOCRLLEY6YSBSWOAQAPHAVXJQQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:13.036416979 (5h57m26.623473231s) [noise=true , address=104.36.82.9:4031 , piece_id=UOVVYSA77DQOK53ATDDXV56PLEVS6GBQ5HMZEKCEURYBPRCXNYPA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:13.172901316 (5h57m26.759957557s) [noise=true , address=104.36.82.47:4544 , piece_id=DVQRASYHRKPFEONZEWJ3MBVM7O5764NPWYADLMG4ARAH5PBD55MA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:13.202754351 (5h57m26.789810592s) [noise=true , address=104.36.82.14:4243 , piece_id=WKT5D56ZXVSJJHVR3GB6N5XR6CGU4VW2XK2YTMPBZOH4S2ZPOVCA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: <nil> Aug 7 22:24:13.202797742 (5h57m26.789853984s) [seg_pos={0 0}, upload=6089]: Segment reached optimal threshold of 14 pieces. Aug 7 22:24:13.203007333 (5h57m26.790063585s) [noise=true , address=104.36.82.58:4280 , piece_id=W2A65H4FIUOCUSMSVCANR2OPVBUFO4YNAFZV3KCASSCSPCHONGBA, seg_pos={0 0}, upload=6089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fgHaEFKK4CeGg4x3vywTuYfT3kmac31xqg2QpsahQNXmvjhW8): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:13.203190164 (5h57m26.790246415s) [noise=true , address=104.36.82.59:4333 , piece_id=JONL6XDWEUYMHVDPJVB54PE5HCYWN7KRO3FSNAMPJWCEP6QBKYVQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K7U6NXcB6SFBtQFrG7DXt2skACr3u5DX2QsrnToi8EKywS9TN): protocol: write tcp 5.161.89.252:63730->104.36.82.59:4333: use of closed network connection; write tcp 5.161.89.252:63730->104.36.82.59:4333: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63730->104.36.82.59:4333: use of closed network connection Aug 7 22:24:13.203221153 (5h57m26.790277394s) [noise=true , address=104.36.82.45:4921 , piece_id=DCEBY7YTNAO4UR6YZ5LC5W27INFXZQ26BPJ7BHPMXY26P5BDL23Q, seg_pos={0 0}, upload=6089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bBRKXiuSN2cw6KrdP8q4Dsf4MiFCUSgMEGjqG2dKL9RLTQra1): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:13.203424783 (5h57m26.790481023s) [noise=true , address=104.36.82.62:4411 , piece_id=3TMITFZMXF5ESOCJNLFU6EIT7CFUYULYZ6FJUQ6X7T2WBEQMKWFQ, seg_pos={0 0}, upload=6089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: context canceled; context canceled Aug 7 22:24:13.203452675 (5h57m26.790508916s) [seg_pos={0 0}, upload=6089]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:24:13.203591695 (5h57m26.790647935s) [upload=6089]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:13.203602313 (5h57m26.790658565s) [upload=6089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:03.852878645 (6h2m17.439934866s) [upload=6089]: Done uploading segments... Aug 7 22:29:03.852904053 (6h2m17.439960294s) [upload=6089]: Upload finished. err: metaclient: context deadline exceeded Duration: 997.852207ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXX.X.XXXX.............................] duration: 227.225843ms FAIL (2025-08-07 22:50:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 227.251651ms core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 109.94257ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:50:24 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 109.962557ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXX..X...........................] duration: 1.812867462s FAIL (2025-08-07 22:50:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.812894212s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXX..XX..XX.............................] duration: 262.107957ms FAIL (2025-08-07 22:50:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 262.135368ms edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 6 out of 60 is failed (threshold=3) [..........................XX.XXXX...........................] duration: 34.402256ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:50:16 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 34.428787ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXX..XXXX.X..........................] duration: 946.394324ms FAIL (2025-08-07 22:50:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 946.414051ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......................XXXXXXXX.X............................] duration: 1.818558384s FAIL (2025-08-07 22:50:11 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.818588309s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXX..X..........................] duration: 1.617477006s FAIL (2025-08-07 22:50:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.617494959s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 22 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXX.XXXXX.X..........................] duration: 181.655111ms FAIL (2025-08-07 22:49:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 420501s) [upload=6055]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:06.462569787 (5h55m20.049626008s) [upload=6061]: Starting upload Aug 7 22:22:06.462655577 (5h55m20.049711788s) [upload=6061]: Done starting upload Aug 7 22:22:06.462750936 (5h55m20.049807157s) [upload=6061]: Uploading segments... Aug 7 22:22:06.462782464 (5h55m20.049838675s) [upload=6061]: Got next segment. Inline: true Aug 7 22:22:06.462786181 (5h55m20.049842402s) [upload=6061]: Waiting for error group managing segments... Aug 7 22:22:06.462817791 (5h55m20.049874001s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:06.462821648 (5h55m20.049877869s) [upload=6061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:06.462823982 (5h55m20.049880203s) [upload=6061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:36.449657257 (5h55m50.036713498s) [upload=6061]: Done uploading segments... Aug 7 22:22:36.449678196 (5h55m50.036734448s) [upload=6061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:59.911739779 (5h56m13.498795989s) [upload=6078]: Starting upload Aug 7 22:22:59.911786736 (5h56m13.498842957s) [upload=6078]: Done starting upload Aug 7 22:22:59.911827472 (5h56m13.498883683s) [upload=6078]: Uploading segments... Aug 7 22:22:59.911861074 (5h56m13.498917295s) [upload=6078]: Got next segment. Inline: true Aug 7 22:22:59.911864401 (5h56m13.498920622s) [upload=6078]: Waiting for error group managing segments... Aug 7 22:22:59.911889107 (5h56m13.498945319s) [upload=6078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:59.911893005 (5h56m13.498949226s) [upload=6078]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:59.911894898 (5h56m13.498951119s) [upload=6078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:29.900458747 (5h56m43.487514958s) [upload=6078]: Done uploading segments... Aug 7 22:23:29.900473835 (5h56m43.487530047s) [upload=6078]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:58.370463254 (5h57m11.957519465s) [upload=6094]: Starting upload Aug 7 22:23:58.370512907 (5h57m11.957569128s) [upload=6094]: Done starting upload Aug 7 22:23:58.370553031 (5h57m11.957609243s) [upload=6094]: Uploading segments... Aug 7 22:23:58.370578639 (5h57m11.95763486s) [upload=6094]: Got next segment. Inline: true Aug 7 22:23:58.370582427 (5h57m11.957638648s) [upload=6094]: Waiting for error group managing segments... Aug 7 22:23:58.370610951 (5h57m11.957667172s) [upload=6094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:58.370615178 (5h57m11.957671399s) [upload=6094]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:58.370617392 (5h57m11.957673613s) [upload=6094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:58.450972793 (5h57m12.038029004s) [upload=6094]: Done uploading segments... Aug 7 22:23:58.450986078 (5h57m12.038042289s) [upload=6094]: Upload finished. err: <nil> Aug 7 22:24:40.794473113 (5h57m54.381529325s) [upload=6107]: Starting upload Aug 7 22:24:40.794523428 (5h57m54.381579639s) [upload=6107]: Done starting upload Aug 7 22:24:40.794565206 (5h57m54.381621417s) [upload=6107]: Uploading segments... Aug 7 22:24:40.794591815 (5h57m54.381648026s) [upload=6107]: Got next segment. Inline: true Aug 7 22:24:40.794595622 (5h57m54.381651833s) [upload=6107]: Waiting for error group managing segments... Aug 7 22:24:40.794624367 (5h57m54.381680577s) [upload=6107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:40.794628323 (5h57m54.381684535s) [upload=6107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:40.794630638 (5h57m54.381686849s) [upload=6107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:10.774549311 (5h58m24.361605533s) [upload=6107]: Done uploading segments... Aug 7 22:25:10.774570019 (5h58m24.361626231s) [upload=6107]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:10.855666099 (5h58m24.442722339s) [upload=6114]: Starting upload Aug 7 22:25:10.855752680 (5h58m24.442808921s) [upload=6114]: Done starting upload Aug 7 22:25:10.855822740 (5h58m24.442878991s) [upload=6114]: Uploading segments... Aug 7 22:25:10.855866683 (5h58m24.442922923s) [upload=6114]: Got next segment. Inline: true Aug 7 22:25:10.855872564 (5h58m24.442928805s) [upload=6114]: Waiting for error group managing segments... Aug 7 22:25:10.855966008 (5h58m24.443022249s) [upload=6114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:10.855972971 (5h58m24.443029212s) [upload=6114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:10.855976568 (5h58m24.443032819s) [upload=6114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:40.842457902 (5h58m54.429514153s) [upload=6114]: Done uploading segments... Aug 7 22:25:40.842479021 (5h58m54.429535272s) [upload=6114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:40.940807525 (5h58m54.527863766s) [upload=6122]: Starting upload Aug 7 22:25:40.940896992 (5h58m54.527953233s) [upload=6122]: Done starting upload Aug 7 22:25:40.940964388 (5h58m54.528020628s) [upload=6122]: Uploading segments... Aug 7 22:25:40.941010434 (5h58m54.528066675s) [upload=6122]: Got next segment. Inline: true Aug 7 22:25:40.941016375 (5h58m54.528072616s) [upload=6122]: Waiting for error group managing segments... Aug 7 22:25:40.941060899 (5h58m54.52811715s) [upload=6122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:40.941068091 (5h58m54.528124332s) [upload=6122]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:40.941073060 (5h58m54.528129311s) [upload=6122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:10.929528068 (5h59m24.516584319s) [upload=6122]: Done uploading segments... Aug 7 22:26:10.929548557 (5h59m24.516604797s) [upload=6122]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:11.008231659 (5h59m24.59528788s) [upload=6129]: Starting upload Aug 7 22:26:11.008286501 (5h59m24.595342713s) [upload=6129]: Done starting upload Aug 7 22:26:11.008333198 (5h59m24.595389419s) [upload=6129]: Uploading segments... Aug 7 22:26:11.008364838 (5h59m24.595421059s) [upload=6129]: Got next segment. Inline: true Aug 7 22:26:11.008368595 (5h59m24.595424816s) [upload=6129]: Waiting for error group managing segments... Aug 7 22:26:11.008399943 (5h59m24.595456164s) [upload=6129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:11.008403599 (5h59m24.59545981s) [upload=6129]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:11.008406565 (5h59m24.595462786s) [upload=6129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:40.996305184 (5h59m54.583361385s) [upload=6129]: Done uploading segments... Aug 7 22:26:40.996323769 (5h59m54.58337998s) [upload=6129]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:14.829073707 (6h0m28.416129948s) [upload=6144]: Starting upload Aug 7 22:27:14.829153145 (6h0m28.416209386s) [upload=6144]: Done starting upload Aug 7 22:27:14.829247050 (6h0m28.416303291s) [upload=6144]: Uploading segments... Aug 7 22:27:14.829299899 (6h0m28.41635614s) [upload=6144]: Got next segment. Inline: true Aug 7 22:27:14.829306702 (6h0m28.416362943s) [upload=6144]: Waiting for error group managing segments... Aug 7 22:27:14.829350864 (6h0m28.416407105s) [upload=6144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:14.829357416 (6h0m28.416413657s) [upload=6144]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:14.829361143 (6h0m28.416417384s) [upload=6144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:44.808549878 (6h0m58.395606099s) [upload=6144]: Done uploading segments... Aug 7 22:27:44.808564877 (6h0m58.395621098s) [upload=6144]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:23.619901331 (6h1m37.206957573s) [upload=6162]: Starting upload Aug 7 22:28:23.620017008 (6h1m37.207073248s) [upload=6162]: Done starting upload Aug 7 22:28:23.620072551 (6h1m37.207128791s) [upload=6162]: Uploading segments... Aug 7 22:28:23.620125059 (6h1m37.2071813s) [upload=6162]: Got next segment. Inline: true Aug 7 22:28:23.620132432 (6h1m37.207188672s) [upload=6162]: Waiting for error group managing segments... Aug 7 22:28:23.620198335 (6h1m37.207254577s) [upload=6162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:23.620204407 (6h1m37.207260647s) [upload=6162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:23.620208153 (6h1m37.207264395s) [upload=6162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:23.683041871 (6h1m37.270098112s) [upload=6162]: Done uploading segments... Aug 7 22:28:23.683055537 (6h1m37.270111778s) [upload=6162]: Upload finished. err: <nil> Aug 7 22:28:40.804037302 (6h1m54.391093523s) [upload=6165]: Starting upload Aug 7 22:28:40.804089460 (6h1m54.391145671s) [upload=6165]: Done starting upload Aug 7 22:28:40.804134294 (6h1m54.391190515s) [upload=6165]: Uploading segments... Aug 7 22:28:40.804181422 (6h1m54.391237632s) [upload=6165]: Got next segment. Inline: true Aug 7 22:28:40.804185379 (6h1m54.3912416s) [upload=6165]: Waiting for error group managing segments... Aug 7 22:28:40.804217198 (6h1m54.391273419s) [upload=6165]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:40.804220915 (6h1m54.391277127s) [upload=6165]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:40.804223370 (6h1m54.391279591s) [upload=6165]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:10.793781791 (6h2m24.380838023s) [upload=6165]: Done uploading segments... Aug 7 22:29:10.793803392 (6h2m24.380859603s) [upload=6165]: Upload finished. err: metaclient: context deadline exceeded Duration: 181.675368ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXX.XXX................................] duration: 551.60496ms FAIL (2025-08-07 22:49:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== TVRWLFHTOF2YZFDYX3LU5FENJAS6JOW572GLJ2OHVQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.749054955 (5h56m13.336111176s) [noise=true , address=104.36.82.2:4730 , piece_id=FKFZNBUC7L2WVTTFPNRSPJUWW6HAEE6CNUAYSHVDYXW5XENS462Q, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.749426248 (5h56m13.336482459s) [noise=true , address=45.91.116.9:4262 , piece_id=R7J2NVZJTXR7ATH2VUC75B2Z73TSJAQUBYJ3WE75RQ6AD3L3PSCQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.750681361 (5h56m13.337737582s) [noise=true , address=104.36.82.5:4840 , piece_id=6SYENOMYMIEYCEFSHJKMXOQE4RCDPDWBAHYDVRTXCQYWJGQDCCKA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.751093369 (5h56m13.338149581s) [noise=true , address=104.36.82.14:4221 , piece_id=SBKEVRBOXXBFOVTZWPSDVDPXEDV6HZIVAV4MWFIYDMNEU643KBWA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.751643326 (5h56m13.338699547s) [noise=true , address=45.91.116.4:4940 , piece_id=UXKWVJQC6KYUHGXKHRYHJATOP5KQF4BY2LWNQJQEGFYKQEBT6GWA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.752249137 (5h56m13.339305348s) [noise=true , address=77.90.31.13:4628 , piece_id=BYQWTXNUD76XIWZZAUZNHDUX7F7CABMDCPK7M7QPDHS3CFBRBMVQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.752826465 (5h56m13.339882676s) [noise=true , address=104.36.82.2:4731 , piece_id=L6W56WDLHKGHEPXBK7F2OYZ6CRM66ZVKG5RGJPH3FHC7RZZ3QGBA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.753473372 (5h56m13.340529583s) [noise=true , address=45.91.116.12:4227 , piece_id=7RIB2VCODFSHTRTBGI3Z74IECO3ATQXSXW7CY3TCA43MZKSKEQYA, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.753961884 (5h56m13.341018096s) [noise=true , address=77.90.31.10:4519 , piece_id=QF2SVPCWB5HG4S7GQL6NFHTYPVOHP4HHORRY2TMJSRXP5A3J43TQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.754616727 (5h56m13.341672948s) [noise=true , address=104.36.82.9:4006 , piece_id=MDXFW3M6TBQJXYHMBTPIOD5I5PK3NUHDRZ3EERNCRKFVMUPSW43Q, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.755122451 (5h56m13.342178672s) [noise=true , address=104.36.82.8:4971 , piece_id=2MZZBGEPYJSHOBAG4FUAU5NKKAOL72C2AWD3GZECQXGPMOG6TDMQ, seg_pos={0 0}, upload=6077]: Uploading piece... Aug 7 22:22:59.837536198 (5h56m13.424592419s) [upload=6077]: Next returned nil segment Aug 7 22:22:59.837547418 (5h56m13.424603629s) [upload=6077]: Waiting for error group managing segments... Aug 7 22:22:59.846810959 (5h56m13.43386718s) [noise=true , address=45.91.116.7:4011 , piece_id=E7YGVXQXIO2A34B22AHGKBQXER4KLEX3AFGHVFUQ6USLQMUFSW2Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.847279324 (5h56m13.434335535s) [noise=true , address=45.91.116.12:4227 , piece_id=7RIB2VCODFSHTRTBGI3Z74IECO3ATQXSXW7CY3TCA43MZKSKEQYA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.848559454 (5h56m13.435615675s) [noise=true , address=45.91.116.4:4940 , piece_id=UXKWVJQC6KYUHGXKHRYHJATOP5KQF4BY2LWNQJQEGFYKQEBT6GWA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.852382881 (5h56m13.439439102s) [noise=true , address=45.91.116.9:4262 , piece_id=R7J2NVZJTXR7ATH2VUC75B2Z73TSJAQUBYJ3WE75RQ6AD3L3PSCQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.852577974 (5h56m13.439634195s) [noise=true , address=77.90.31.19:4842 , piece_id=ULCHXICRO3ULX7IEW6GVXHHGXT6F42N3WQDEEUVOSKCXID6F6XZQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.856642091 (5h56m13.443698302s) [noise=true , address=104.36.82.8:4971 , piece_id=2MZZBGEPYJSHOBAG4FUAU5NKKAOL72C2AWD3GZECQXGPMOG6TDMQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.856766534 (5h56m13.443822745s) [noise=true , address=77.90.31.10:4519 , piece_id=QF2SVPCWB5HG4S7GQL6NFHTYPVOHP4HHORRY2TMJSRXP5A3J43TQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.868085922 (5h56m13.455142143s) [noise=true , address=45.91.116.12:4225 , piece_id=MNJGBICBIG3HQXCO7B2C2ITRFSOW475RAGNVTAQNVFQYAJ7DOU5Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.868246762 (5h56m13.455302973s) [noise=true , address=104.36.82.2:4730 , piece_id=FKFZNBUC7L2WVTTFPNRSPJUWW6HAEE6CNUAYSHVDYXW5XENS462Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.877804453 (5h56m13.464860674s) [noise=true , address=104.36.82.11:4115 , piece_id=BALIRX6S5WW2CGSXSIFHWV2ABDH3ASPHR36YEYRUYTTGZCW6U6AQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.882357252 (5h56m13.469413462s) [noise=true , address=104.36.82.2:4731 , piece_id=L6W56WDLHKGHEPXBK7F2OYZ6CRM66ZVKG5RGJPH3FHC7RZZ3QGBA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.887398722 (5h56m13.474454933s) [noise=true , address=104.36.82.14:4221 , piece_id=SBKEVRBOXXBFOVTZWPSDVDPXEDV6HZIVAV4MWFIYDMNEU643KBWA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.887674137 (5h56m13.474730358s) [noise=true , address=104.36.82.10:4079 , piece_id=DHLJEMGMQUTVRWLFHTOF2YZFDYX3LU5FENJAS6JOW572GLJ2OHVQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: <nil> Aug 7 22:22:59.887689665 (5h56m13.474745877s) [seg_pos={0 0}, upload=6077]: Segment reached optimal threshold of 13 pieces. Aug 7 22:22:59.887775385 (5h56m13.474831606s) [noise=true , address=77.90.31.13:4628 , piece_id=BYQWTXNUD76XIWZZAUZNHDUX7F7CABMDCPK7M7QPDHS3CFBRBMVQ, seg_pos={0 0}, upload=6077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Aug 7 22:22:59.887795453 (5h56m13.474851674s) [noise=true , address=104.36.82.5:4840 , piece_id=6SYENOMYMIEYCEFSHJKMXOQE4RCDPDWBAHYDVRTXCQYWJGQDCCKA, seg_pos={0 0}, upload=6077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Aug 7 22:22:59.887879469 (5h56m13.47493569s) [noise=true , address=104.36.82.9:4006 , piece_id=MDXFW3M6TBQJXYHMBTPIOD5I5PK3NUHDRZ3EERNCRKFVMUPSW43Q, seg_pos={0 0}, upload=6077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 5.161.89.252:59712->104.36.82.9:4006: use of closed network connection; write tcp 5.161.89.252:59712->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:59712->104.36.82.9:4006: use of closed network connection Aug 7 22:22:59.887943669 (5h56m13.47499988s) [seg_pos={0 0}, upload=6077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:22:59.888006888 (5h56m13.475063109s) [upload=6077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:22:59.888010665 (5h56m13.475066886s) [upload=6077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:59.933305033 (5h56m13.520361243s) [upload=6077]: Done uploading segments... Aug 7 22:22:59.933317746 (5h56m13.520373958s) [upload=6077]: Upload finished. err: <nil> Aug 7 22:23:47.739190890 (5h57m1.326247131s) [upload=6091]: Starting upload Aug 7 22:23:47.739297178 (5h57m1.326353419s) [upload=6091]: Done starting upload Aug 7 22:23:47.739428053 (5h57m1.326484274s) [upload=6091]: Uploading segments... Aug 7 22:23:47.739777785 (5h57m1.326834025s) [upload=6091]: Got next segment. Inline: false Aug 7 22:23:47.739805196 (5h57m1.326861407s) [upload=6091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:47.739811638 (5h57m1.326867849s) [upload=6091]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:17.722308743 (5h57m31.309364985s) [upload=6091]: Next segment err: context deadline exceeded Aug 7 22:24:17.722438996 (5h57m31.309495238s) [upload=6091]: Done uploading segments... Aug 7 22:24:17.722476005 (5h57m31.309532246s) [upload=6091]: Upload finished. err: context deadline exceeded Aug 7 22:24:18.676382471 (5h57m32.263438711s) [upload=6101]: Starting upload Aug 7 22:24:18.676482739 (5h57m32.263538979s) [upload=6101]: Done starting upload Aug 7 22:24:18.676619723 (5h57m32.263675964s) [upload=6101]: Uploading segments... Aug 7 22:24:18.676886642 (5h57m32.263942883s) [upload=6101]: Got next segment. Inline: false Aug 7 22:24:18.676922940 (5h57m32.263979181s) [upload=6101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:18.676930835 (5h57m32.263987076s) [upload=6101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:48.662510024 (5h58m2.249566255s) [upload=6101]: Next segment err: context deadline exceeded Aug 7 22:24:48.662708235 (5h58m2.249764465s) [upload=6101]: Done uploading segments... Aug 7 22:24:48.662735284 (5h58m2.249791526s) [upload=6101]: Upload finished. err: context deadline exceeded Aug 7 22:25:18.142081018 (5h58m31.729137259s) [upload=6117]: Starting upload Aug 7 22:25:18.142160586 (5h58m31.729216828s) [upload=6117]: Done starting upload Aug 7 22:25:18.142286381 (5h58m31.729342623s) [upload=6117]: Uploading segments... Aug 7 22:25:18.142594587 (5h58m31.729650828s) [upload=6117]: Got next segment. Inline: false Aug 7 22:25:18.142641995 (5h58m31.729698236s) [upload=6117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:18.142647436 (5h58m31.729703676s) [upload=6117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:25:48.125202589 (5h59m1.71225884s) [upload=6117]: Next segment err: context deadline exceeded Aug 7 22:25:48.125308446 (5h59m1.712364688s) [upload=6117]: Done uploading segments... Aug 7 22:25:48.125355273 (5h59m1.712411525s) [upload=6117]: Upload finished. err: context deadline exceeded Duration: 551.626129ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXX..XXXX.X.........................] duration: 444.998625ms FAIL (2025-08-07 22:49:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 445.014855ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 228.867561ms FAIL (2025-08-07 22:49:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 228.882298ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXX..X.........................] duration: 1.503813274s FAIL (2025-08-07 22:49:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.503834384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......................XXXXXXXX.X...........................] duration: 1.937181239s FAIL (2025-08-07 22:49:41 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.937211645s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 14 out of 60 is failed (threshold=3) [............XXXXXXXXXXX.X.XX................................] duration: 190.612973ms FAIL (2025-08-07 22:49:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:04.377299713 (5h52m17.964355924s) [upload=6018]: Starting upload Aug 7 22:19:04.377350166 (5h52m17.964406387s) [upload=6018]: Done starting upload Aug 7 22:19:04.377401523 (5h52m17.964457744s) [upload=6018]: Uploading segments... Aug 7 22:19:04.377446897 (5h52m17.964503118s) [upload=6018]: Got next segment. Inline: true Aug 7 22:19:04.377453410 (5h52m17.96450962s) [upload=6018]: Waiting for error group managing segments... Aug 7 22:19:04.377487142 (5h52m17.964543354s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:04.377491361 (5h52m17.964547582s) [upload=6018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:04.377493875 (5h52m17.964550086s) [upload=6018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:04.361964803 (5h53m17.949021024s) [upload=6018]: Done uploading segments... Aug 7 22:20:04.361979240 (5h53m17.949035461s) [upload=6018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:11.847632631 (5h53m25.434688843s) [upload=6035]: Starting upload Aug 7 22:20:11.847694537 (5h53m25.434750747s) [upload=6035]: Done starting upload Aug 7 22:20:11.847740192 (5h53m25.434796403s) [upload=6035]: Uploading segments... Aug 7 22:20:11.847772231 (5h53m25.434828442s) [upload=6035]: Got next segment. Inline: true Aug 7 22:20:11.847776860 (5h53m25.434833071s) [upload=6035]: Waiting for error group managing segments... Aug 7 22:20:11.847824679 (5h53m25.434880891s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:11.847830130 (5h53m25.434886341s) [upload=6035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:11.847833205 (5h53m25.434889417s) [upload=6035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:11.836000640 (5h54m25.42305688s) [upload=6035]: Done uploading segments... Aug 7 22:21:11.836026087 (5h54m25.423082328s) [upload=6035]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:11.965120248 (5h54m25.552176489s) [upload=6050]: Starting upload Aug 7 22:21:11.965231515 (5h54m25.552287757s) [upload=6050]: Done starting upload Aug 7 22:21:11.965301156 (5h54m25.552357397s) [upload=6050]: Uploading segments... Aug 7 22:21:11.965348053 (5h54m25.552404304s) [upload=6050]: Got next segment. Inline: true Aug 7 22:21:11.965353994 (5h54m25.552410235s) [upload=6050]: Waiting for error group managing segments... Aug 7 22:21:11.965399049 (5h54m25.55245529s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:11.965405170 (5h54m25.552461411s) [upload=6050]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:11.965408957 (5h54m25.552465198s) [upload=6050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:11.952333050 (5h55m25.539389261s) [upload=6050]: Done uploading segments... Aug 7 22:22:11.952348108 (5h55m25.539404329s) [upload=6050]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:12.125652873 (5h55m25.712709094s) [upload=6063]: Starting upload Aug 7 22:22:12.125702476 (5h55m25.712758688s) [upload=6063]: Done starting upload Aug 7 22:22:12.125747020 (5h55m25.712803241s) [upload=6063]: Uploading segments... Aug 7 22:22:12.125776895 (5h55m25.712833116s) [upload=6063]: Got next segment. Inline: true Aug 7 22:22:12.125781404 (5h55m25.712837625s) [upload=6063]: Waiting for error group managing segments... Aug 7 22:22:12.125814315 (5h55m25.712870536s) [upload=6063]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:12.125818052 (5h55m25.712874263s) [upload=6063]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:12.125820397 (5h55m25.712876618s) [upload=6063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:12.024254277 (5h56m25.611310498s) [upload=6063]: Done uploading segments... Aug 7 22:23:12.024289072 (5h56m25.611345283s) [upload=6063]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:12.175471355 (5h56m25.762527616s) [upload=6082]: Starting upload Aug 7 22:23:12.175573175 (5h56m25.762629466s) [upload=6082]: Done starting upload Aug 7 22:23:12.175670356 (5h56m25.762726627s) [upload=6082]: Uploading segments... Aug 7 22:23:12.175760093 (5h56m25.762816364s) [upload=6082]: Got next segment. Inline: true Aug 7 22:23:12.175767677 (5h56m25.762823919s) [upload=6082]: Waiting for error group managing segments... Aug 7 22:23:12.175819345 (5h56m25.762875585s) [upload=6082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:12.175825105 (5h56m25.762881346s) [upload=6082]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:12.175828732 (5h56m25.762884973s) [upload=6082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:12.154559214 (5h57m25.741615455s) [upload=6082]: Done uploading segments... Aug 7 22:24:12.154578851 (5h57m25.741635092s) [upload=6082]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:12.253523032 (5h57m25.840579253s) [upload=6098]: Starting upload Aug 7 22:24:12.253572454 (5h57m25.840628665s) [upload=6098]: Done starting upload Aug 7 22:24:12.253615034 (5h57m25.840671245s) [upload=6098]: Uploading segments... Aug 7 22:24:12.253654808 (5h57m25.840711029s) [upload=6098]: Got next segment. Inline: true Aug 7 22:24:12.253658636 (5h57m25.840714847s) [upload=6098]: Waiting for error group managing segments... Aug 7 22:24:12.253689122 (5h57m25.840745343s) [upload=6098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.253692969 (5h57m25.84074918s) [upload=6098]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:12.253695303 (5h57m25.840751524s) [upload=6098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:12.467343237 (5h57m26.054399448s) [upload=6098]: Done uploading segments... Aug 7 22:24:12.467365428 (5h57m26.05442164s) [upload=6098]: Upload finished. err: <nil> Aug 7 22:24:33.085790933 (5h57m46.672847184s) [upload=6106]: Starting upload Aug 7 22:24:33.085870281 (5h57m46.672926522s) [upload=6106]: Done starting upload Aug 7 22:24:33.085932668 (5h57m46.67298892s) [upload=6106]: Uploading segments... Aug 7 22:24:33.085994053 (5h57m46.673050293s) [upload=6106]: Got next segment. Inline: true Aug 7 22:24:33.086000064 (5h57m46.673056304s) [upload=6106]: Waiting for error group managing segments... Aug 7 22:24:33.086045709 (5h57m46.67310195s) [upload=6106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:33.086051168 (5h57m46.67310741s) [upload=6106]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:33.086054985 (5h57m46.673111227s) [upload=6106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:33.072711385 (5h58m46.659767636s) [upload=6106]: Done uploading segments... Aug 7 22:25:33.072753384 (5h58m46.659809634s) [upload=6106]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:33.160012555 (5h58m46.747068795s) [upload=6120]: Starting upload Aug 7 22:25:33.160101981 (5h58m46.747158222s) [upload=6120]: Done starting upload Aug 7 22:25:33.160199473 (5h58m46.747255715s) [upload=6120]: Uploading segments... Aug 7 22:25:33.160243596 (5h58m46.747299837s) [upload=6120]: Got next segment. Inline: true Aug 7 22:25:33.160249046 (5h58m46.747305287s) [upload=6120]: Waiting for error group managing segments... Aug 7 22:25:33.160294231 (5h58m46.747350482s) [upload=6120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:33.160300012 (5h58m46.747356252s) [upload=6120]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:33.160303619 (5h58m46.747359859s) [upload=6120]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:33.228188865 (5h58m46.815245086s) [upload=6120]: Done uploading segments... Aug 7 22:25:33.228202571 (5h58m46.815258782s) [upload=6120]: Upload finished. err: <nil> Aug 7 22:26:14.972993755 (5h59m28.560049967s) [upload=6132]: Starting upload Aug 7 22:26:14.973047095 (5h59m28.560103307s) [upload=6132]: Done starting upload Aug 7 22:26:14.973101456 (5h59m28.560157677s) [upload=6132]: Uploading segments... Aug 7 22:26:14.973137704 (5h59m28.560193925s) [upload=6132]: Got next segment. Inline: true Aug 7 22:26:14.973141382 (5h59m28.560197592s) [upload=6132]: Waiting for error group managing segments... Aug 7 22:26:14.973187948 (5h59m28.560244158s) [upload=6132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:14.973191995 (5h59m28.560248217s) [upload=6132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:14.973195201 (5h59m28.560251423s) [upload=6132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:14.950324988 (6h0m28.537381229s) [upload=6132]: Done uploading segments... Aug 7 22:27:14.950345787 (6h0m28.537402028s) [upload=6132]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:22.925128081 (6h0m36.512184292s) [upload=6146]: Starting upload Aug 7 22:27:22.925194244 (6h0m36.512250465s) [upload=6146]: Done starting upload Aug 7 22:27:22.925235571 (6h0m36.512291783s) [upload=6146]: Uploading segments... Aug 7 22:27:22.925260508 (6h0m36.512316729s) [upload=6146]: Got next segment. Inline: true Aug 7 22:27:22.925264486 (6h0m36.512320707s) [upload=6146]: Waiting for error group managing segments... Aug 7 22:27:22.925305752 (6h0m36.512361973s) [upload=6146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:22.925309750 (6h0m36.512365971s) [upload=6146]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:22.925312044 (6h0m36.512368265s) [upload=6146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:22.913115749 (6h1m36.50017199s) [upload=6146]: Done uploading segments... Aug 7 22:28:22.913136929 (6h1m36.500193169s) [upload=6146]: Upload finished. err: metaclient: context deadline exceeded Duration: 190.635645ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXX..XX..XX............................] duration: 308.227261ms FAIL (2025-08-07 22:49:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 308.253911ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)