Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [....................................................X.XXXXX.] duration: 240.173414ms FAIL (2025-08-07 23:20:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 240.198591ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 188.315289ms FAIL (2025-08-07 23:19:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 188.328793ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 19.231667ms target: https://us1.storj.io/login FAIL (2025-08-07 23:19:57 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 19.277023ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................X.XXXXX] duration: 30.001385491s FAIL (2025-08-07 23:19:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00142793s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXXXX..] duration: 1m0.001749058s FAIL (2025-08-07 23:19:47 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001767312s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................................................XXXXXX.X.X.] duration: 1.850780926s FAIL (2025-08-07 23:19:43 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.850811894s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.................................................XXXXX.XX...] duration: 165.994294ms FAIL (2025-08-07 23:19:42 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:04.749315126 (6h47m18.336371367s) [upload=6964]: Starting upload Aug 7 23:14:04.749394674 (6h47m18.336450915s) [upload=6964]: Done starting upload Aug 7 23:14:04.749474242 (6h47m18.336530484s) [upload=6964]: Uploading segments... Aug 7 23:14:04.749532461 (6h47m18.336588702s) [upload=6964]: Got next segment. Inline: true Aug 7 23:14:04.749538312 (6h47m18.336594553s) [upload=6964]: Waiting for error group managing segments... Aug 7 23:14:04.749584738 (6h47m18.33664098s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:04.749590058 (6h47m18.3366463s) [upload=6964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:04.749593755 (6h47m18.336649997s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:34.737068324 (6h47m48.324124536s) [upload=6964]: Done uploading segments... Aug 7 23:14:34.737089104 (6h47m48.324145314s) [upload=6964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.277141003 (6h47m50.864197244s) [upload=6974]: Starting upload Aug 7 23:14:37.277320197 (6h47m50.864376439s) [upload=6974]: Done starting upload Aug 7 23:14:37.277421095 (6h47m50.864477337s) [upload=6974]: Uploading segments... Aug 7 23:14:37.277519849 (6h47m50.864576091s) [upload=6974]: Got next segment. Inline: true Aug 7 23:14:37.277527635 (6h47m50.864583875s) [upload=6974]: Waiting for error group managing segments... Aug 7 23:14:37.277578299 (6h47m50.86463454s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.277584181 (6h47m50.864640421s) [upload=6974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.277587807 (6h47m50.864644049s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.263997045 (6h48m20.851053287s) [upload=6974]: Done uploading segments... Aug 7 23:15:07.264036289 (6h48m20.85109253s) [upload=6974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:40.767391975 (6h48m54.354448226s) [upload=6993]: Starting upload Aug 7 23:15:40.767481722 (6h48m54.354537963s) [upload=6993]: Done starting upload Aug 7 23:15:40.767544320 (6h48m54.35460057s) [upload=6993]: Uploading segments... Aug 7 23:15:40.767585375 (6h48m54.354641617s) [upload=6993]: Got next segment. Inline: true Aug 7 23:15:40.767591397 (6h48m54.354647638s) [upload=6993]: Waiting for error group managing segments... Aug 7 23:15:40.767637002 (6h48m54.354693244s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:40.767642423 (6h48m54.354698664s) [upload=6993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:40.767646530 (6h48m54.354702782s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:10.754586120 (6h49m24.341642341s) [upload=6993]: Done uploading segments... Aug 7 23:16:10.754601419 (6h49m24.34165763s) [upload=6993]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:26.719060301 (6h49m40.306116542s) [upload=7005]: Starting upload Aug 7 23:16:26.719141191 (6h49m40.306197433s) [upload=7005]: Done starting upload Aug 7 23:16:26.719210051 (6h49m40.306266292s) [upload=7005]: Uploading segments... Aug 7 23:16:26.719285351 (6h49m40.306341592s) [upload=7005]: Got next segment. Inline: true Aug 7 23:16:26.719292685 (6h49m40.306348926s) [upload=7005]: Waiting for error group managing segments... Aug 7 23:16:26.719340825 (6h49m40.306397066s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:26.719347277 (6h49m40.306403518s) [upload=7005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:26.719351064 (6h49m40.306407305s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:26.802515608 (6h49m40.389571849s) [upload=7005]: Done uploading segments... Aug 7 23:16:26.802535986 (6h49m40.389592227s) [upload=7005]: Upload finished. err: <nil> Aug 7 23:16:42.191377275 (6h49m55.778433516s) [upload=7011]: Starting upload Aug 7 23:16:42.191466421 (6h49m55.778522662s) [upload=7011]: Done starting upload Aug 7 23:16:42.191531582 (6h49m55.778587803s) [upload=7011]: Uploading segments... Aug 7 23:16:42.191607003 (6h49m55.778663224s) [upload=7011]: Got next segment. Inline: true Aug 7 23:16:42.191612414 (6h49m55.778668635s) [upload=7011]: Waiting for error group managing segments... Aug 7 23:16:42.191651346 (6h49m55.778707556s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:42.191656575 (6h49m55.778712787s) [upload=7011]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:42.191659481 (6h49m55.778715702s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.173345160 (6h50m25.760401401s) [upload=7011]: Done uploading segments... Aug 7 23:17:12.173364576 (6h50m25.760420816s) [upload=7011]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:12.261509015 (6h50m25.848565257s) [upload=7016]: Starting upload Aug 7 23:17:12.261597881 (6h50m25.848654122s) [upload=7016]: Done starting upload Aug 7 23:17:12.261665168 (6h50m25.848721419s) [upload=7016]: Uploading segments... Aug 7 23:17:12.261708137 (6h50m25.848764388s) [upload=7016]: Got next segment. Inline: true Aug 7 23:17:12.261713487 (6h50m25.848769728s) [upload=7016]: Waiting for error group managing segments... Aug 7 23:17:12.261755746 (6h50m25.848811988s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:12.261761196 (6h50m25.848817448s) [upload=7016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:12.261764593 (6h50m25.848820834s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:42.241895867 (6h50m55.828952078s) [upload=7016]: Done uploading segments... Aug 7 23:17:42.241913931 (6h50m55.828970141s) [upload=7016]: Upload finished. err: metaclient: context deadline exceeded Duration: 166.017737ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 413.673102ms FAIL (2025-08-07 23:19:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 413.698758ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................................................XXXXXX.X.X] duration: 30.001430729s FAIL (2025-08-07 23:19:37 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001450987s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [..................................................XXXXX.XX..] duration: 312.921667ms FAIL (2025-08-07 23:19:23 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:04.749315126 (6h47m18.336371367s) [upload=6964]: Starting upload Aug 7 23:14:04.749394674 (6h47m18.336450915s) [upload=6964]: Done starting upload Aug 7 23:14:04.749474242 (6h47m18.336530484s) [upload=6964]: Uploading segments... Aug 7 23:14:04.749532461 (6h47m18.336588702s) [upload=6964]: Got next segment. Inline: true Aug 7 23:14:04.749538312 (6h47m18.336594553s) [upload=6964]: Waiting for error group managing segments... Aug 7 23:14:04.749584738 (6h47m18.33664098s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:04.749590058 (6h47m18.3366463s) [upload=6964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:04.749593755 (6h47m18.336649997s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:34.737068324 (6h47m48.324124536s) [upload=6964]: Done uploading segments... Aug 7 23:14:34.737089104 (6h47m48.324145314s) [upload=6964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.277141003 (6h47m50.864197244s) [upload=6974]: Starting upload Aug 7 23:14:37.277320197 (6h47m50.864376439s) [upload=6974]: Done starting upload Aug 7 23:14:37.277421095 (6h47m50.864477337s) [upload=6974]: Uploading segments... Aug 7 23:14:37.277519849 (6h47m50.864576091s) [upload=6974]: Got next segment. Inline: true Aug 7 23:14:37.277527635 (6h47m50.864583875s) [upload=6974]: Waiting for error group managing segments... Aug 7 23:14:37.277578299 (6h47m50.86463454s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.277584181 (6h47m50.864640421s) [upload=6974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.277587807 (6h47m50.864644049s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.263997045 (6h48m20.851053287s) [upload=6974]: Done uploading segments... Aug 7 23:15:07.264036289 (6h48m20.85109253s) [upload=6974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:40.767391975 (6h48m54.354448226s) [upload=6993]: Starting upload Aug 7 23:15:40.767481722 (6h48m54.354537963s) [upload=6993]: Done starting upload Aug 7 23:15:40.767544320 (6h48m54.35460057s) [upload=6993]: Uploading segments... Aug 7 23:15:40.767585375 (6h48m54.354641617s) [upload=6993]: Got next segment. Inline: true Aug 7 23:15:40.767591397 (6h48m54.354647638s) [upload=6993]: Waiting for error group managing segments... Aug 7 23:15:40.767637002 (6h48m54.354693244s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:40.767642423 (6h48m54.354698664s) [upload=6993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:40.767646530 (6h48m54.354702782s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:10.754586120 (6h49m24.341642341s) [upload=6993]: Done uploading segments... Aug 7 23:16:10.754601419 (6h49m24.34165763s) [upload=6993]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:26.719060301 (6h49m40.306116542s) [upload=7005]: Starting upload Aug 7 23:16:26.719141191 (6h49m40.306197433s) [upload=7005]: Done starting upload Aug 7 23:16:26.719210051 (6h49m40.306266292s) [upload=7005]: Uploading segments... Aug 7 23:16:26.719285351 (6h49m40.306341592s) [upload=7005]: Got next segment. Inline: true Aug 7 23:16:26.719292685 (6h49m40.306348926s) [upload=7005]: Waiting for error group managing segments... Aug 7 23:16:26.719340825 (6h49m40.306397066s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:26.719347277 (6h49m40.306403518s) [upload=7005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:26.719351064 (6h49m40.306407305s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:26.802515608 (6h49m40.389571849s) [upload=7005]: Done uploading segments... Aug 7 23:16:26.802535986 (6h49m40.389592227s) [upload=7005]: Upload finished. err: <nil> Aug 7 23:16:42.191377275 (6h49m55.778433516s) [upload=7011]: Starting upload Aug 7 23:16:42.191466421 (6h49m55.778522662s) [upload=7011]: Done starting upload Aug 7 23:16:42.191531582 (6h49m55.778587803s) [upload=7011]: Uploading segments... Aug 7 23:16:42.191607003 (6h49m55.778663224s) [upload=7011]: Got next segment. Inline: true Aug 7 23:16:42.191612414 (6h49m55.778668635s) [upload=7011]: Waiting for error group managing segments... Aug 7 23:16:42.191651346 (6h49m55.778707556s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:42.191656575 (6h49m55.778712787s) [upload=7011]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:42.191659481 (6h49m55.778715702s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.173345160 (6h50m25.760401401s) [upload=7011]: Done uploading segments... Aug 7 23:17:12.173364576 (6h50m25.760420816s) [upload=7011]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:12.261509015 (6h50m25.848565257s) [upload=7016]: Starting upload Aug 7 23:17:12.261597881 (6h50m25.848654122s) [upload=7016]: Done starting upload Aug 7 23:17:12.261665168 (6h50m25.848721419s) [upload=7016]: Uploading segments... Aug 7 23:17:12.261708137 (6h50m25.848764388s) [upload=7016]: Got next segment. Inline: true Aug 7 23:17:12.261713487 (6h50m25.848769728s) [upload=7016]: Waiting for error group managing segments... Aug 7 23:17:12.261755746 (6h50m25.848811988s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:12.261761196 (6h50m25.848817448s) [upload=7016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:12.261764593 (6h50m25.848820834s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:42.241895867 (6h50m55.828952078s) [upload=7016]: Done uploading segments... Aug 7 23:17:42.241913931 (6h50m55.828970141s) [upload=7016]: Upload finished. err: metaclient: context deadline exceeded Duration: 312.94478ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 213.729404ms FAIL (2025-08-07 23:19:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 213.751506ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 815.55452ms FAIL (2025-08-07 23:19:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 815.581382ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [......................................................X.XXXX] duration: 30.001136968s FAIL (2025-08-07 23:19:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001254708s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXXX.X..] duration: 3.285951169s FAIL (2025-08-07 23:19:13 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.285970876s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................X.XX.X....] duration: 2.366655831s FAIL (2025-08-07 23:19:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.366732313s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.227961006s FAIL (2025-08-07 23:19:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 1.227984669s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 465.251523ms FAIL (2025-08-07 23:19:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 465.270449ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 13.385635ms target: https://us1.storj.io/login FAIL (2025-08-07 23:18:56 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 13.405782ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXXXX.] duration: 44.162622021s FAIL (2025-08-07 23:18:47 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.162672375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXX.X.] duration: 622.843005ms FAIL (2025-08-07 23:18:46 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== NEIQLWSFHH6VIHQ, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.245596960 (6h50m14.832653171s) [noise=true , address=104.36.82.38:4722 , piece_id=KAKNFXRVST57ER7OSMJQPLWP5WPRSAUO2XGIR5NBNYSKKLEVHHKQ, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.245792406 (6h50m14.832848627s) [noise=true , address=104.36.82.43:4837 , piece_id=PLDDTFAQ46JT2FULOU4NZ2DTWLCNFLHQ4C3MRNY6SSBD632F7PTA, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.245877034 (6h50m14.832933255s) [noise=true , address=104.36.82.7:4931 , piece_id=QUXCDFRAHLWS5F7VHUSVOJW6HDOZEA3RCXS73TU7HB437JELVWWQ, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246000935 (6h50m14.833057156s) [noise=true , address=104.36.82.15:4280 , piece_id=4X4RRASLES7QJI7QLVXHAK6D25ESTENVCP4QTIAKFUOXHLLFEASA, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246161564 (6h50m14.833217785s) [noise=true , address=45.91.116.11:4138 , piece_id=XTJ72N5QFBFI6SMCBMDVQMH5MDVESNZNQRJ3O2RYBE5YMZKUH5PQ, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246211418 (6h50m14.833267639s) [noise=true , address=104.36.82.58:4291 , piece_id=G4WDWWTPTW52QQWPN5LZOOT6K2H7TPC4LM74VOLC4MU4IYV5NSXA, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246439463 (6h50m14.833495684s) [noise=true , address=104.36.82.62:4426 , piece_id=NRTMQDX47Z62O6HM53XZ3V37JKZ6OSNKF5RTP743QZILV2544KYQ, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246486741 (6h50m14.833542982s) [noise=true , address=104.36.82.16:4306 , piece_id=4EOWRYBPUCSKSCOP6F2MVAWTCCRSSN52LUBQ2PHUQ2WD65ZZRL6A, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246645107 (6h50m14.833701318s) [noise=true , address=104.36.82.44:4869 , piece_id=5HEQMPK5FFZT7LEQBO2FP4K5WFZUUXZFOVNL7YFD33EDHBGFM5LA, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246768127 (6h50m14.833824348s) [noise=true , address=104.36.82.34:4579 , piece_id=REYZ6VVM5FF7UY5O2LNKY4KQSUZ6UMXI6SUBJHSGEYMKNUJDT22Q, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246851282 (6h50m14.833907503s) [noise=true , address=104.36.82.39:4759 , piece_id=PFGRM2FMVS3QYE4HINLEEF67JOCER4RVZMSF4MTR6PM4AIEBLVWA, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246984430 (6h50m14.834040651s) [noise=true , address=104.36.82.53:4150 , piece_id=WHULIMGAKVHVYXOMM6OAXTHKBSOFOMKYQBJIOGJ6V2Q6DZHLNKUQ, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.247295290 (6h50m14.834351502s) [noise=true , address=45.91.116.10:4113 , piece_id=D6IJCK7MM5HRW57NPTVRN4KHFAOQ3PCHOZ6GKYD7R5CHBVM3QY5Q, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.247669469 (6h50m14.8347257s) [noise=true , address=104.36.82.46:4511 , piece_id=ULRNJR6YENNPT6J3WKYYMOS6TGSYYCZWXCXJCNT7MTSNQTIAYA4A, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.250694777 (6h50m14.837750987s) [noise=true , address=64.49.13.155:4923 , piece_id=IYYQMAQC2WYMFNAN2QJKX6TOOASYDMFXLQW733YJPH36KRTTWCUA, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.336573307 (6h50m14.923629548s) [upload=7012]: Next returned nil segment Aug 7 23:17:01.336675598 (6h50m14.923731829s) [upload=7012]: Waiting for error group managing segments... Aug 7 23:17:01.349864215 (6h50m14.936920427s) [noise=true , address=104.36.82.46:4511 , piece_id=ULRNJR6YENNPT6J3WKYYMOS6TGSYYCZWXCXJCNT7MTSNQTIAYA4A, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.350635474 (6h50m14.937691696s) [noise=true , address=104.36.82.16:4306 , piece_id=4EOWRYBPUCSKSCOP6F2MVAWTCCRSSN52LUBQ2PHUQ2WD65ZZRL6A, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.351508765 (6h50m14.938564976s) [noise=true , address=104.36.82.7:4931 , piece_id=QUXCDFRAHLWS5F7VHUSVOJW6HDOZEA3RCXS73TU7HB437JELVWWQ, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.361648412 (6h50m14.948704633s) [noise=true , address=45.91.116.11:4138 , piece_id=XTJ72N5QFBFI6SMCBMDVQMH5MDVESNZNQRJ3O2RYBE5YMZKUH5PQ, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.372296818 (6h50m14.959353039s) [noise=true , address=104.36.82.38:4722 , piece_id=KAKNFXRVST57ER7OSMJQPLWP5WPRSAUO2XGIR5NBNYSKKLEVHHKQ, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.374412216 (6h50m14.961468437s) [noise=true , address=104.36.82.53:4150 , piece_id=WHULIMGAKVHVYXOMM6OAXTHKBSOFOMKYQBJIOGJ6V2Q6DZHLNKUQ, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.376334315 (6h50m14.963390536s) [noise=true , address=104.36.82.9:4036 , piece_id=4JXETZVM5JSPKNOOADOYASPNQPVNHYWPXQ3H6NDSHTPWGD4QSHLA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.377151810 (6h50m14.964208021s) [noise=true , address=104.36.82.39:4759 , piece_id=PFGRM2FMVS3QYE4HINLEEF67JOCER4RVZMSF4MTR6PM4AIEBLVWA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.381030461 (6h50m14.968086682s) [noise=true , address=104.36.82.58:4291 , piece_id=G4WDWWTPTW52QQWPN5LZOOT6K2H7TPC4LM74VOLC4MU4IYV5NSXA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.381246624 (6h50m14.968302845s) [noise=true , address=45.91.116.10:4113 , piece_id=D6IJCK7MM5HRW57NPTVRN4KHFAOQ3PCHOZ6GKYD7R5CHBVM3QY5Q, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.386284257 (6h50m14.973340478s) [noise=true , address=104.36.82.34:4579 , piece_id=REYZ6VVM5FF7UY5O2LNKY4KQSUZ6UMXI6SUBJHSGEYMKNUJDT22Q, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.393562553 (6h50m14.980618774s) [noise=true , address=104.36.82.15:4280 , piece_id=4X4RRASLES7QJI7QLVXHAK6D25ESTENVCP4QTIAKFUOXHLLFEASA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.522590225 (6h50m15.109646465s) [noise=true , address=104.36.82.61:4386 , piece_id=MXEPTE6DAIWGRKW3QKEBNKLDZLBGFDFDV3LJHNEIQLWSFHH6VIHQ, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.618970397 (6h50m15.206026608s) [noise=true , address=104.36.82.44:4869 , piece_id=5HEQMPK5FFZT7LEQBO2FP4K5WFZUUXZFOVNL7YFD33EDHBGFM5LA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.618997969 (6h50m15.20605418s) [seg_pos={0 0}, upload=7012]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:01.619194586 (6h50m15.206250807s) [noise=true , address=104.36.82.43:4837 , piece_id=PLDDTFAQ46JT2FULOU4NZ2DTWLCNFLHQ4C3MRNY6SSBD632F7PTA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jmioLYNGZjDcByx8B1oHbqfid2DRqRfgoHhz85VcoTqbWz6Ls): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:01.619320160 (6h50m15.206376371s) [noise=true , address=104.36.82.62:4426 , piece_id=NRTMQDX47Z62O6HM53XZ3V37JKZ6OSNKF5RTP743QZILV2544KYQ, seg_pos={0 0}, upload=7012]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BySTQsT3jUKEyGKfyQxSHYCxYf8XYiZsooiNPMECH9QhYJVQV): protocol: write tcp 5.161.89.252:40900->104.36.82.62:4426: use of closed network connection; write tcp 5.161.89.252:40900->104.36.82.62:4426: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:40900->104.36.82.62:4426: use of closed network connection Aug 7 23:17:01.619437519 (6h50m15.206493731s) [noise=true , address=104.36.82.59:4315 , piece_id=OAL4COHVWB5S7HYXSST5ZPK72UGJILVDQYGMCF3VD5VR3IQDBY7A, seg_pos={0 0}, upload=7012]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1o5mMqSq2uht75TtSp9i4NpnkpA3AbwmNG9ffyp5zZehUqcMuz): protocol: read tcp 5.161.89.252:35802->104.36.82.59:4315: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:01.619456385 (6h50m15.206512595s) [noise=true , address=64.49.13.155:4923 , piece_id=IYYQMAQC2WYMFNAN2QJKX6TOOASYDMFXLQW733YJPH36KRTTWCUA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: write tcp 5.161.89.252:57632->64.49.13.155:4923: use of closed network connection; write tcp 5.161.89.252:57632->64.49.13.155:4923: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:57632->64.49.13.155:4923: use of closed network connection Aug 7 23:17:01.619489577 (6h50m15.206545788s) [seg_pos={0 0}, upload=7012]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:01.619578854 (6h50m15.206635075s) [upload=7012]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:01.619583262 (6h50m15.206639483s) [upload=7012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:01.664125864 (6h50m15.251182076s) [upload=7012]: Done uploading segments... Aug 7 23:17:01.664141804 (6h50m15.251198016s) [upload=7012]: Upload finished. err: <nil> Aug 7 23:17:46.010046338 (6h50m59.597102549s) [upload=7025]: Starting upload Aug 7 23:17:46.010097403 (6h50m59.597153624s) [upload=7025]: Done starting upload Aug 7 23:17:46.010160732 (6h50m59.597216943s) [upload=7025]: Uploading segments... Aug 7 23:17:46.010323555 (6h50m59.597379767s) [upload=7025]: Got next segment. Inline: false Aug 7 23:17:46.010347600 (6h50m59.597403821s) [upload=7025]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:46.010351267 (6h50m59.597407488s) [upload=7025]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:45.997132272 (6h51m59.584188484s) [upload=7025]: Next segment err: context deadline exceeded Aug 7 23:18:45.997256144 (6h51m59.584312365s) [upload=7025]: Done uploading segments... Aug 7 23:18:45.997274097 (6h51m59.584330309s) [upload=7025]: Upload finished. err: context deadline exceeded Duration: 622.863493ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................X.XX.X...] duration: 529.9485ms FAIL (2025-08-07 23:18:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 529.966383ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................X.XXX] duration: 30.000647205s FAIL (2025-08-07 23:18:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000662683s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................XXX.X] duration: 1m0.000363117s FAIL (2025-08-07 23:18:45 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== NEIQLWSFHH6VIHQ, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.245596960 (6h50m14.832653171s) [noise=true , address=104.36.82.38:4722 , piece_id=KAKNFXRVST57ER7OSMJQPLWP5WPRSAUO2XGIR5NBNYSKKLEVHHKQ, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.245792406 (6h50m14.832848627s) [noise=true , address=104.36.82.43:4837 , piece_id=PLDDTFAQ46JT2FULOU4NZ2DTWLCNFLHQ4C3MRNY6SSBD632F7PTA, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.245877034 (6h50m14.832933255s) [noise=true , address=104.36.82.7:4931 , piece_id=QUXCDFRAHLWS5F7VHUSVOJW6HDOZEA3RCXS73TU7HB437JELVWWQ, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246000935 (6h50m14.833057156s) [noise=true , address=104.36.82.15:4280 , piece_id=4X4RRASLES7QJI7QLVXHAK6D25ESTENVCP4QTIAKFUOXHLLFEASA, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246161564 (6h50m14.833217785s) [noise=true , address=45.91.116.11:4138 , piece_id=XTJ72N5QFBFI6SMCBMDVQMH5MDVESNZNQRJ3O2RYBE5YMZKUH5PQ, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246211418 (6h50m14.833267639s) [noise=true , address=104.36.82.58:4291 , piece_id=G4WDWWTPTW52QQWPN5LZOOT6K2H7TPC4LM74VOLC4MU4IYV5NSXA, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246439463 (6h50m14.833495684s) [noise=true , address=104.36.82.62:4426 , piece_id=NRTMQDX47Z62O6HM53XZ3V37JKZ6OSNKF5RTP743QZILV2544KYQ, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246486741 (6h50m14.833542982s) [noise=true , address=104.36.82.16:4306 , piece_id=4EOWRYBPUCSKSCOP6F2MVAWTCCRSSN52LUBQ2PHUQ2WD65ZZRL6A, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246645107 (6h50m14.833701318s) [noise=true , address=104.36.82.44:4869 , piece_id=5HEQMPK5FFZT7LEQBO2FP4K5WFZUUXZFOVNL7YFD33EDHBGFM5LA, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246768127 (6h50m14.833824348s) [noise=true , address=104.36.82.34:4579 , piece_id=REYZ6VVM5FF7UY5O2LNKY4KQSUZ6UMXI6SUBJHSGEYMKNUJDT22Q, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246851282 (6h50m14.833907503s) [noise=true , address=104.36.82.39:4759 , piece_id=PFGRM2FMVS3QYE4HINLEEF67JOCER4RVZMSF4MTR6PM4AIEBLVWA, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.246984430 (6h50m14.834040651s) [noise=true , address=104.36.82.53:4150 , piece_id=WHULIMGAKVHVYXOMM6OAXTHKBSOFOMKYQBJIOGJ6V2Q6DZHLNKUQ, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.247295290 (6h50m14.834351502s) [noise=true , address=45.91.116.10:4113 , piece_id=D6IJCK7MM5HRW57NPTVRN4KHFAOQ3PCHOZ6GKYD7R5CHBVM3QY5Q, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.247669469 (6h50m14.8347257s) [noise=true , address=104.36.82.46:4511 , piece_id=ULRNJR6YENNPT6J3WKYYMOS6TGSYYCZWXCXJCNT7MTSNQTIAYA4A, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.250694777 (6h50m14.837750987s) [noise=true , address=64.49.13.155:4923 , piece_id=IYYQMAQC2WYMFNAN2QJKX6TOOASYDMFXLQW733YJPH36KRTTWCUA, seg_pos={0 0}, upload=7012]: Uploading piece... Aug 7 23:17:01.336573307 (6h50m14.923629548s) [upload=7012]: Next returned nil segment Aug 7 23:17:01.336675598 (6h50m14.923731829s) [upload=7012]: Waiting for error group managing segments... Aug 7 23:17:01.349864215 (6h50m14.936920427s) [noise=true , address=104.36.82.46:4511 , piece_id=ULRNJR6YENNPT6J3WKYYMOS6TGSYYCZWXCXJCNT7MTSNQTIAYA4A, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.350635474 (6h50m14.937691696s) [noise=true , address=104.36.82.16:4306 , piece_id=4EOWRYBPUCSKSCOP6F2MVAWTCCRSSN52LUBQ2PHUQ2WD65ZZRL6A, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.351508765 (6h50m14.938564976s) [noise=true , address=104.36.82.7:4931 , piece_id=QUXCDFRAHLWS5F7VHUSVOJW6HDOZEA3RCXS73TU7HB437JELVWWQ, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.361648412 (6h50m14.948704633s) [noise=true , address=45.91.116.11:4138 , piece_id=XTJ72N5QFBFI6SMCBMDVQMH5MDVESNZNQRJ3O2RYBE5YMZKUH5PQ, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.372296818 (6h50m14.959353039s) [noise=true , address=104.36.82.38:4722 , piece_id=KAKNFXRVST57ER7OSMJQPLWP5WPRSAUO2XGIR5NBNYSKKLEVHHKQ, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.374412216 (6h50m14.961468437s) [noise=true , address=104.36.82.53:4150 , piece_id=WHULIMGAKVHVYXOMM6OAXTHKBSOFOMKYQBJIOGJ6V2Q6DZHLNKUQ, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.376334315 (6h50m14.963390536s) [noise=true , address=104.36.82.9:4036 , piece_id=4JXETZVM5JSPKNOOADOYASPNQPVNHYWPXQ3H6NDSHTPWGD4QSHLA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.377151810 (6h50m14.964208021s) [noise=true , address=104.36.82.39:4759 , piece_id=PFGRM2FMVS3QYE4HINLEEF67JOCER4RVZMSF4MTR6PM4AIEBLVWA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.381030461 (6h50m14.968086682s) [noise=true , address=104.36.82.58:4291 , piece_id=G4WDWWTPTW52QQWPN5LZOOT6K2H7TPC4LM74VOLC4MU4IYV5NSXA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.381246624 (6h50m14.968302845s) [noise=true , address=45.91.116.10:4113 , piece_id=D6IJCK7MM5HRW57NPTVRN4KHFAOQ3PCHOZ6GKYD7R5CHBVM3QY5Q, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.386284257 (6h50m14.973340478s) [noise=true , address=104.36.82.34:4579 , piece_id=REYZ6VVM5FF7UY5O2LNKY4KQSUZ6UMXI6SUBJHSGEYMKNUJDT22Q, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.393562553 (6h50m14.980618774s) [noise=true , address=104.36.82.15:4280 , piece_id=4X4RRASLES7QJI7QLVXHAK6D25ESTENVCP4QTIAKFUOXHLLFEASA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.522590225 (6h50m15.109646465s) [noise=true , address=104.36.82.61:4386 , piece_id=MXEPTE6DAIWGRKW3QKEBNKLDZLBGFDFDV3LJHNEIQLWSFHH6VIHQ, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.618970397 (6h50m15.206026608s) [noise=true , address=104.36.82.44:4869 , piece_id=5HEQMPK5FFZT7LEQBO2FP4K5WFZUUXZFOVNL7YFD33EDHBGFM5LA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: <nil> Aug 7 23:17:01.618997969 (6h50m15.20605418s) [seg_pos={0 0}, upload=7012]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:01.619194586 (6h50m15.206250807s) [noise=true , address=104.36.82.43:4837 , piece_id=PLDDTFAQ46JT2FULOU4NZ2DTWLCNFLHQ4C3MRNY6SSBD632F7PTA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jmioLYNGZjDcByx8B1oHbqfid2DRqRfgoHhz85VcoTqbWz6Ls): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:01.619320160 (6h50m15.206376371s) [noise=true , address=104.36.82.62:4426 , piece_id=NRTMQDX47Z62O6HM53XZ3V37JKZ6OSNKF5RTP743QZILV2544KYQ, seg_pos={0 0}, upload=7012]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BySTQsT3jUKEyGKfyQxSHYCxYf8XYiZsooiNPMECH9QhYJVQV): protocol: write tcp 5.161.89.252:40900->104.36.82.62:4426: use of closed network connection; write tcp 5.161.89.252:40900->104.36.82.62:4426: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:40900->104.36.82.62:4426: use of closed network connection Aug 7 23:17:01.619437519 (6h50m15.206493731s) [noise=true , address=104.36.82.59:4315 , piece_id=OAL4COHVWB5S7HYXSST5ZPK72UGJILVDQYGMCF3VD5VR3IQDBY7A, seg_pos={0 0}, upload=7012]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1o5mMqSq2uht75TtSp9i4NpnkpA3AbwmNG9ffyp5zZehUqcMuz): protocol: read tcp 5.161.89.252:35802->104.36.82.59:4315: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:01.619456385 (6h50m15.206512595s) [noise=true , address=64.49.13.155:4923 , piece_id=IYYQMAQC2WYMFNAN2QJKX6TOOASYDMFXLQW733YJPH36KRTTWCUA, seg_pos={0 0}, upload=7012]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: write tcp 5.161.89.252:57632->64.49.13.155:4923: use of closed network connection; write tcp 5.161.89.252:57632->64.49.13.155:4923: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:57632->64.49.13.155:4923: use of closed network connection Aug 7 23:17:01.619489577 (6h50m15.206545788s) [seg_pos={0 0}, upload=7012]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:01.619578854 (6h50m15.206635075s) [upload=7012]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:01.619583262 (6h50m15.206639483s) [upload=7012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:01.664125864 (6h50m15.251182076s) [upload=7012]: Done uploading segments... Aug 7 23:17:01.664141804 (6h50m15.251198016s) [upload=7012]: Upload finished. err: <nil> Aug 7 23:17:46.010046338 (6h50m59.597102549s) [upload=7025]: Starting upload Aug 7 23:17:46.010097403 (6h50m59.597153624s) [upload=7025]: Done starting upload Aug 7 23:17:46.010160732 (6h50m59.597216943s) [upload=7025]: Uploading segments... Aug 7 23:17:46.010323555 (6h50m59.597379767s) [upload=7025]: Got next segment. Inline: false Aug 7 23:17:46.010347600 (6h50m59.597403821s) [upload=7025]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:46.010351267 (6h50m59.597407488s) [upload=7025]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:45.997132272 (6h51m59.584188484s) [upload=7025]: Next segment err: context deadline exceeded Aug 7 23:18:45.997256144 (6h51m59.584312365s) [upload=7025]: Done uploading segments... Aug 7 23:18:45.997274097 (6h51m59.584330309s) [upload=7025]: Upload finished. err: context deadline exceeded Duration: 1m0.00038112s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XXXXXX.] duration: 274.766298ms FAIL (2025-08-07 23:18:34 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=6888]: Done uploading segments... Aug 7 23:09:43.008607604 (6h42m56.595663825s) [upload=6888]: Upload finished. err: <nil> Aug 7 23:10:07.090790160 (6h43m20.677846371s) [upload=6895]: Starting upload Aug 7 23:10:07.090866883 (6h43m20.677923094s) [upload=6895]: Done starting upload Aug 7 23:10:07.090916526 (6h43m20.677972737s) [upload=6895]: Uploading segments... Aug 7 23:10:07.091175900 (6h43m20.678232142s) [upload=6895]: Got next segment. Inline: true Aug 7 23:10:07.091186920 (6h43m20.678243141s) [upload=6895]: Waiting for error group managing segments... Aug 7 23:10:07.091251632 (6h43m20.678307853s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:07.091257803 (6h43m20.678314014s) [upload=6895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:07.091262211 (6h43m20.678318423s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:07.150936870 (6h43m20.737993091s) [upload=6895]: Done uploading segments... Aug 7 23:10:07.150955715 (6h43m20.738011936s) [upload=6895]: Upload finished. err: <nil> Aug 7 23:10:35.935324537 (6h43m49.522380777s) [upload=6905]: Starting upload Aug 7 23:10:35.935406759 (6h43m49.522463001s) [upload=6905]: Done starting upload Aug 7 23:10:35.935481139 (6h43m49.52253739s) [upload=6905]: Uploading segments... Aug 7 23:10:35.935518358 (6h43m49.522574599s) [upload=6905]: Got next segment. Inline: true Aug 7 23:10:35.935523989 (6h43m49.522580229s) [upload=6905]: Waiting for error group managing segments... Aug 7 23:10:35.935567379 (6h43m49.522623621s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:35.935572860 (6h43m49.522629101s) [upload=6905]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:35.935576386 (6h43m49.522632628s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.018368185 (6h43m49.605424426s) [upload=6905]: Done uploading segments... Aug 7 23:10:36.018385587 (6h43m49.605441827s) [upload=6905]: Upload finished. err: <nil> Aug 7 23:10:57.038963659 (6h44m10.62601988s) [upload=6913]: Starting upload Aug 7 23:10:57.039017149 (6h44m10.62607337s) [upload=6913]: Done starting upload Aug 7 23:10:57.039065159 (6h44m10.62612137s) [upload=6913]: Uploading segments... Aug 7 23:10:57.039093831 (6h44m10.626150042s) [upload=6913]: Got next segment. Inline: true Aug 7 23:10:57.039097308 (6h44m10.626153519s) [upload=6913]: Waiting for error group managing segments... Aug 7 23:10:57.039142663 (6h44m10.626198874s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:57.039146320 (6h44m10.626202531s) [upload=6913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:57.039148564 (6h44m10.626204785s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:57.092133855 (6h44m10.679190086s) [upload=6913]: Done uploading segments... Aug 7 23:10:57.092152090 (6h44m10.67920833s) [upload=6913]: Upload finished. err: <nil> Aug 7 23:11:47.690012005 (6h45m1.277068256s) [upload=6926]: Starting upload Aug 7 23:11:47.690127641 (6h45m1.277183882s) [upload=6926]: Done starting upload Aug 7 23:11:47.690277249 (6h45m1.27733349s) [upload=6926]: Uploading segments... Aug 7 23:11:47.690316773 (6h45m1.277373025s) [upload=6926]: Got next segment. Inline: true Aug 7 23:11:47.690360005 (6h45m1.277416245s) [upload=6926]: Waiting for error group managing segments... Aug 7 23:11:47.690405779 (6h45m1.277462031s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:47.690411880 (6h45m1.277468122s) [upload=6926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:47.690415508 (6h45m1.277471748s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:47.740982296 (6h45m1.328038537s) [upload=6926]: Done uploading segments... Aug 7 23:11:47.740995380 (6h45m1.328051622s) [upload=6926]: Upload finished. err: <nil> Aug 7 23:12:15.388374517 (6h45m28.975430749s) [upload=6935]: Starting upload Aug 7 23:12:15.388464206 (6h45m28.975520437s) [upload=6935]: Done starting upload Aug 7 23:12:15.388550818 (6h45m28.975607039s) [upload=6935]: Uploading segments... Aug 7 23:12:15.388651916 (6h45m28.975708127s) [upload=6935]: Got next segment. Inline: true Aug 7 23:12:15.388660222 (6h45m28.975716443s) [upload=6935]: Waiting for error group managing segments... Aug 7 23:12:15.388716747 (6h45m28.975772968s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:15.388723729 (6h45m28.97577995s) [upload=6935]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:15.388727838 (6h45m28.975784059s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:15.375897715 (6h46m28.962953947s) [upload=6935]: Done uploading segments... Aug 7 23:13:15.375942599 (6h46m28.962998841s) [upload=6935]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:15.452799710 (6h46m29.039855931s) [upload=6951]: Starting upload Aug 7 23:13:15.452856376 (6h46m29.039912597s) [upload=6951]: Done starting upload Aug 7 23:13:15.452901079 (6h46m29.0399573s) [upload=6951]: Uploading segments... Aug 7 23:13:15.452938739 (6h46m29.03999495s) [upload=6951]: Got next segment. Inline: true Aug 7 23:13:15.452943217 (6h46m29.039999439s) [upload=6951]: Waiting for error group managing segments... Aug 7 23:13:15.452974977 (6h46m29.040031188s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:15.452979305 (6h46m29.040035526s) [upload=6951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:15.452982110 (6h46m29.040038331s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:15.442587501 (6h47m29.029643742s) [upload=6951]: Done uploading segments... Aug 7 23:14:15.442608670 (6h47m29.029664922s) [upload=6951]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:15.516163727 (6h47m29.103219978s) [upload=6968]: Starting upload Aug 7 23:14:15.516314939 (6h47m29.10337124s) [upload=6968]: Done starting upload Aug 7 23:14:15.516411479 (6h47m29.103467721s) [upload=6968]: Uploading segments... Aug 7 23:14:15.516477332 (6h47m29.103533573s) [upload=6968]: Got next segment. Inline: true Aug 7 23:14:15.516487672 (6h47m29.103543913s) [upload=6968]: Waiting for error group managing segments... Aug 7 23:14:15.516556882 (6h47m29.103613162s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:15.516567169 (6h47m29.103623411s) [upload=6968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:15.516573491 (6h47m29.103629733s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:15.501845849 (6h48m29.08890207s) [upload=6968]: Done uploading segments... Aug 7 23:15:15.501865025 (6h48m29.088921247s) [upload=6968]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:15.591828711 (6h48m29.178884922s) [upload=6986]: Starting upload Aug 7 23:15:15.591883403 (6h48m29.178939613s) [upload=6986]: Done starting upload Aug 7 23:15:15.591959133 (6h48m29.179015344s) [upload=6986]: Uploading segments... Aug 7 23:15:15.592001994 (6h48m29.179058205s) [upload=6986]: Got next segment. Inline: true Aug 7 23:15:15.592022743 (6h48m29.179078964s) [upload=6986]: Waiting for error group managing segments... Aug 7 23:15:15.592060944 (6h48m29.179117156s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:15.592065653 (6h48m29.179121874s) [upload=6986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:15.592067957 (6h48m29.179124178s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:15.578451341 (6h49m29.165507562s) [upload=6986]: Done uploading segments... Aug 7 23:16:15.578469606 (6h49m29.165525816s) [upload=6986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:23.278937820 (6h49m36.865994061s) [upload=7004]: Starting upload Aug 7 23:16:23.279020784 (6h49m36.866077025s) [upload=7004]: Done starting upload Aug 7 23:16:23.279084963 (6h49m36.866141205s) [upload=7004]: Uploading segments... Aug 7 23:16:23.279128285 (6h49m36.866184526s) [upload=7004]: Got next segment. Inline: true Aug 7 23:16:23.279135057 (6h49m36.866191299s) [upload=7004]: Waiting for error group managing segments... Aug 7 23:16:23.279179039 (6h49m36.86623528s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:23.279184509 (6h49m36.866240751s) [upload=7004]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:23.279188327 (6h49m36.866244558s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:23.264696428 (6h50m36.851752649s) [upload=7004]: Done uploading segments... Aug 7 23:17:23.264712168 (6h50m36.851768379s) [upload=7004]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.354485739 (6h50m36.94154196s) [upload=7020]: Starting upload Aug 7 23:17:23.354537225 (6h50m36.941593446s) [upload=7020]: Done starting upload Aug 7 23:17:23.354582019 (6h50m36.94163824s) [upload=7020]: Uploading segments... Aug 7 23:17:23.354609931 (6h50m36.941666142s) [upload=7020]: Got next segment. Inline: true Aug 7 23:17:23.354613586 (6h50m36.941669807s) [upload=7020]: Waiting for error group managing segments... Aug 7 23:17:23.354647150 (6h50m36.941703371s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.354650656 (6h50m36.941706877s) [upload=7020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.354652961 (6h50m36.941709172s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:23.336455483 (6h51m36.923511723s) [upload=7020]: Done uploading segments... Aug 7 23:18:23.336475711 (6h51m36.923531952s) [upload=7020]: Upload finished. err: metaclient: context deadline exceeded Duration: 274.785333ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................................................XXXXXX.X.] duration: 1.820225794s FAIL (2025-08-07 23:18:27 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.820248788s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 1m0.001063885s FAIL (2025-08-07 23:18:23 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=6888]: Done uploading segments... Aug 7 23:09:43.008607604 (6h42m56.595663825s) [upload=6888]: Upload finished. err: <nil> Aug 7 23:10:07.090790160 (6h43m20.677846371s) [upload=6895]: Starting upload Aug 7 23:10:07.090866883 (6h43m20.677923094s) [upload=6895]: Done starting upload Aug 7 23:10:07.090916526 (6h43m20.677972737s) [upload=6895]: Uploading segments... Aug 7 23:10:07.091175900 (6h43m20.678232142s) [upload=6895]: Got next segment. Inline: true Aug 7 23:10:07.091186920 (6h43m20.678243141s) [upload=6895]: Waiting for error group managing segments... Aug 7 23:10:07.091251632 (6h43m20.678307853s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:07.091257803 (6h43m20.678314014s) [upload=6895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:07.091262211 (6h43m20.678318423s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:07.150936870 (6h43m20.737993091s) [upload=6895]: Done uploading segments... Aug 7 23:10:07.150955715 (6h43m20.738011936s) [upload=6895]: Upload finished. err: <nil> Aug 7 23:10:35.935324537 (6h43m49.522380777s) [upload=6905]: Starting upload Aug 7 23:10:35.935406759 (6h43m49.522463001s) [upload=6905]: Done starting upload Aug 7 23:10:35.935481139 (6h43m49.52253739s) [upload=6905]: Uploading segments... Aug 7 23:10:35.935518358 (6h43m49.522574599s) [upload=6905]: Got next segment. Inline: true Aug 7 23:10:35.935523989 (6h43m49.522580229s) [upload=6905]: Waiting for error group managing segments... Aug 7 23:10:35.935567379 (6h43m49.522623621s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:35.935572860 (6h43m49.522629101s) [upload=6905]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:35.935576386 (6h43m49.522632628s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.018368185 (6h43m49.605424426s) [upload=6905]: Done uploading segments... Aug 7 23:10:36.018385587 (6h43m49.605441827s) [upload=6905]: Upload finished. err: <nil> Aug 7 23:10:57.038963659 (6h44m10.62601988s) [upload=6913]: Starting upload Aug 7 23:10:57.039017149 (6h44m10.62607337s) [upload=6913]: Done starting upload Aug 7 23:10:57.039065159 (6h44m10.62612137s) [upload=6913]: Uploading segments... Aug 7 23:10:57.039093831 (6h44m10.626150042s) [upload=6913]: Got next segment. Inline: true Aug 7 23:10:57.039097308 (6h44m10.626153519s) [upload=6913]: Waiting for error group managing segments... Aug 7 23:10:57.039142663 (6h44m10.626198874s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:57.039146320 (6h44m10.626202531s) [upload=6913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:57.039148564 (6h44m10.626204785s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:57.092133855 (6h44m10.679190086s) [upload=6913]: Done uploading segments... Aug 7 23:10:57.092152090 (6h44m10.67920833s) [upload=6913]: Upload finished. err: <nil> Aug 7 23:11:47.690012005 (6h45m1.277068256s) [upload=6926]: Starting upload Aug 7 23:11:47.690127641 (6h45m1.277183882s) [upload=6926]: Done starting upload Aug 7 23:11:47.690277249 (6h45m1.27733349s) [upload=6926]: Uploading segments... Aug 7 23:11:47.690316773 (6h45m1.277373025s) [upload=6926]: Got next segment. Inline: true Aug 7 23:11:47.690360005 (6h45m1.277416245s) [upload=6926]: Waiting for error group managing segments... Aug 7 23:11:47.690405779 (6h45m1.277462031s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:47.690411880 (6h45m1.277468122s) [upload=6926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:47.690415508 (6h45m1.277471748s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:47.740982296 (6h45m1.328038537s) [upload=6926]: Done uploading segments... Aug 7 23:11:47.740995380 (6h45m1.328051622s) [upload=6926]: Upload finished. err: <nil> Aug 7 23:12:15.388374517 (6h45m28.975430749s) [upload=6935]: Starting upload Aug 7 23:12:15.388464206 (6h45m28.975520437s) [upload=6935]: Done starting upload Aug 7 23:12:15.388550818 (6h45m28.975607039s) [upload=6935]: Uploading segments... Aug 7 23:12:15.388651916 (6h45m28.975708127s) [upload=6935]: Got next segment. Inline: true Aug 7 23:12:15.388660222 (6h45m28.975716443s) [upload=6935]: Waiting for error group managing segments... Aug 7 23:12:15.388716747 (6h45m28.975772968s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:15.388723729 (6h45m28.97577995s) [upload=6935]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:15.388727838 (6h45m28.975784059s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:15.375897715 (6h46m28.962953947s) [upload=6935]: Done uploading segments... Aug 7 23:13:15.375942599 (6h46m28.962998841s) [upload=6935]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:15.452799710 (6h46m29.039855931s) [upload=6951]: Starting upload Aug 7 23:13:15.452856376 (6h46m29.039912597s) [upload=6951]: Done starting upload Aug 7 23:13:15.452901079 (6h46m29.0399573s) [upload=6951]: Uploading segments... Aug 7 23:13:15.452938739 (6h46m29.03999495s) [upload=6951]: Got next segment. Inline: true Aug 7 23:13:15.452943217 (6h46m29.039999439s) [upload=6951]: Waiting for error group managing segments... Aug 7 23:13:15.452974977 (6h46m29.040031188s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:15.452979305 (6h46m29.040035526s) [upload=6951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:15.452982110 (6h46m29.040038331s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:15.442587501 (6h47m29.029643742s) [upload=6951]: Done uploading segments... Aug 7 23:14:15.442608670 (6h47m29.029664922s) [upload=6951]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:15.516163727 (6h47m29.103219978s) [upload=6968]: Starting upload Aug 7 23:14:15.516314939 (6h47m29.10337124s) [upload=6968]: Done starting upload Aug 7 23:14:15.516411479 (6h47m29.103467721s) [upload=6968]: Uploading segments... Aug 7 23:14:15.516477332 (6h47m29.103533573s) [upload=6968]: Got next segment. Inline: true Aug 7 23:14:15.516487672 (6h47m29.103543913s) [upload=6968]: Waiting for error group managing segments... Aug 7 23:14:15.516556882 (6h47m29.103613162s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:15.516567169 (6h47m29.103623411s) [upload=6968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:15.516573491 (6h47m29.103629733s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:15.501845849 (6h48m29.08890207s) [upload=6968]: Done uploading segments... Aug 7 23:15:15.501865025 (6h48m29.088921247s) [upload=6968]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:15.591828711 (6h48m29.178884922s) [upload=6986]: Starting upload Aug 7 23:15:15.591883403 (6h48m29.178939613s) [upload=6986]: Done starting upload Aug 7 23:15:15.591959133 (6h48m29.179015344s) [upload=6986]: Uploading segments... Aug 7 23:15:15.592001994 (6h48m29.179058205s) [upload=6986]: Got next segment. Inline: true Aug 7 23:15:15.592022743 (6h48m29.179078964s) [upload=6986]: Waiting for error group managing segments... Aug 7 23:15:15.592060944 (6h48m29.179117156s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:15.592065653 (6h48m29.179121874s) [upload=6986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:15.592067957 (6h48m29.179124178s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:15.578451341 (6h49m29.165507562s) [upload=6986]: Done uploading segments... Aug 7 23:16:15.578469606 (6h49m29.165525816s) [upload=6986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:23.278937820 (6h49m36.865994061s) [upload=7004]: Starting upload Aug 7 23:16:23.279020784 (6h49m36.866077025s) [upload=7004]: Done starting upload Aug 7 23:16:23.279084963 (6h49m36.866141205s) [upload=7004]: Uploading segments... Aug 7 23:16:23.279128285 (6h49m36.866184526s) [upload=7004]: Got next segment. Inline: true Aug 7 23:16:23.279135057 (6h49m36.866191299s) [upload=7004]: Waiting for error group managing segments... Aug 7 23:16:23.279179039 (6h49m36.86623528s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:23.279184509 (6h49m36.866240751s) [upload=7004]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:23.279188327 (6h49m36.866244558s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:23.264696428 (6h50m36.851752649s) [upload=7004]: Done uploading segments... Aug 7 23:17:23.264712168 (6h50m36.851768379s) [upload=7004]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.354485739 (6h50m36.94154196s) [upload=7020]: Starting upload Aug 7 23:17:23.354537225 (6h50m36.941593446s) [upload=7020]: Done starting upload Aug 7 23:17:23.354582019 (6h50m36.94163824s) [upload=7020]: Uploading segments... Aug 7 23:17:23.354609931 (6h50m36.941666142s) [upload=7020]: Got next segment. Inline: true Aug 7 23:17:23.354613586 (6h50m36.941669807s) [upload=7020]: Waiting for error group managing segments... Aug 7 23:17:23.354647150 (6h50m36.941703371s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.354650656 (6h50m36.941706877s) [upload=7020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.354652961 (6h50m36.941709172s) [upload=7020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:23.336455483 (6h51m36.923511723s) [upload=7020]: Done uploading segments... Aug 7 23:18:23.336475711 (6h51m36.923531952s) [upload=7020]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001080888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 705.881045ms FAIL (2025-08-07 23:18:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 705.907243ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.502546646s FAIL (2025-08-07 23:18:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 1.502570219s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................................................X.XX] duration: 30.00105325s FAIL (2025-08-07 23:18:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001067016s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................X.XX.X..] duration: 975.644022ms FAIL (2025-08-07 23:18:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 975.665363ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [...................................................XXXXX.XX.] duration: 423.039515ms FAIL (2025-08-07 23:18:13 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:04.749315126 (6h47m18.336371367s) [upload=6964]: Starting upload Aug 7 23:14:04.749394674 (6h47m18.336450915s) [upload=6964]: Done starting upload Aug 7 23:14:04.749474242 (6h47m18.336530484s) [upload=6964]: Uploading segments... Aug 7 23:14:04.749532461 (6h47m18.336588702s) [upload=6964]: Got next segment. Inline: true Aug 7 23:14:04.749538312 (6h47m18.336594553s) [upload=6964]: Waiting for error group managing segments... Aug 7 23:14:04.749584738 (6h47m18.33664098s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:04.749590058 (6h47m18.3366463s) [upload=6964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:04.749593755 (6h47m18.336649997s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:34.737068324 (6h47m48.324124536s) [upload=6964]: Done uploading segments... Aug 7 23:14:34.737089104 (6h47m48.324145314s) [upload=6964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.277141003 (6h47m50.864197244s) [upload=6974]: Starting upload Aug 7 23:14:37.277320197 (6h47m50.864376439s) [upload=6974]: Done starting upload Aug 7 23:14:37.277421095 (6h47m50.864477337s) [upload=6974]: Uploading segments... Aug 7 23:14:37.277519849 (6h47m50.864576091s) [upload=6974]: Got next segment. Inline: true Aug 7 23:14:37.277527635 (6h47m50.864583875s) [upload=6974]: Waiting for error group managing segments... Aug 7 23:14:37.277578299 (6h47m50.86463454s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.277584181 (6h47m50.864640421s) [upload=6974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.277587807 (6h47m50.864644049s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.263997045 (6h48m20.851053287s) [upload=6974]: Done uploading segments... Aug 7 23:15:07.264036289 (6h48m20.85109253s) [upload=6974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:40.767391975 (6h48m54.354448226s) [upload=6993]: Starting upload Aug 7 23:15:40.767481722 (6h48m54.354537963s) [upload=6993]: Done starting upload Aug 7 23:15:40.767544320 (6h48m54.35460057s) [upload=6993]: Uploading segments... Aug 7 23:15:40.767585375 (6h48m54.354641617s) [upload=6993]: Got next segment. Inline: true Aug 7 23:15:40.767591397 (6h48m54.354647638s) [upload=6993]: Waiting for error group managing segments... Aug 7 23:15:40.767637002 (6h48m54.354693244s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:40.767642423 (6h48m54.354698664s) [upload=6993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:40.767646530 (6h48m54.354702782s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:10.754586120 (6h49m24.341642341s) [upload=6993]: Done uploading segments... Aug 7 23:16:10.754601419 (6h49m24.34165763s) [upload=6993]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:26.719060301 (6h49m40.306116542s) [upload=7005]: Starting upload Aug 7 23:16:26.719141191 (6h49m40.306197433s) [upload=7005]: Done starting upload Aug 7 23:16:26.719210051 (6h49m40.306266292s) [upload=7005]: Uploading segments... Aug 7 23:16:26.719285351 (6h49m40.306341592s) [upload=7005]: Got next segment. Inline: true Aug 7 23:16:26.719292685 (6h49m40.306348926s) [upload=7005]: Waiting for error group managing segments... Aug 7 23:16:26.719340825 (6h49m40.306397066s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:26.719347277 (6h49m40.306403518s) [upload=7005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:26.719351064 (6h49m40.306407305s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:26.802515608 (6h49m40.389571849s) [upload=7005]: Done uploading segments... Aug 7 23:16:26.802535986 (6h49m40.389592227s) [upload=7005]: Upload finished. err: <nil> Aug 7 23:16:42.191377275 (6h49m55.778433516s) [upload=7011]: Starting upload Aug 7 23:16:42.191466421 (6h49m55.778522662s) [upload=7011]: Done starting upload Aug 7 23:16:42.191531582 (6h49m55.778587803s) [upload=7011]: Uploading segments... Aug 7 23:16:42.191607003 (6h49m55.778663224s) [upload=7011]: Got next segment. Inline: true Aug 7 23:16:42.191612414 (6h49m55.778668635s) [upload=7011]: Waiting for error group managing segments... Aug 7 23:16:42.191651346 (6h49m55.778707556s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:42.191656575 (6h49m55.778712787s) [upload=7011]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:42.191659481 (6h49m55.778715702s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.173345160 (6h50m25.760401401s) [upload=7011]: Done uploading segments... Aug 7 23:17:12.173364576 (6h50m25.760420816s) [upload=7011]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:12.261509015 (6h50m25.848565257s) [upload=7016]: Starting upload Aug 7 23:17:12.261597881 (6h50m25.848654122s) [upload=7016]: Done starting upload Aug 7 23:17:12.261665168 (6h50m25.848721419s) [upload=7016]: Uploading segments... Aug 7 23:17:12.261708137 (6h50m25.848764388s) [upload=7016]: Got next segment. Inline: true Aug 7 23:17:12.261713487 (6h50m25.848769728s) [upload=7016]: Waiting for error group managing segments... Aug 7 23:17:12.261755746 (6h50m25.848811988s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:12.261761196 (6h50m25.848817448s) [upload=7016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:12.261764593 (6h50m25.848820834s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:42.241895867 (6h50m55.828952078s) [upload=7016]: Done uploading segments... Aug 7 23:17:42.241913931 (6h50m55.828970141s) [upload=7016]: Upload finished. err: metaclient: context deadline exceeded Duration: 423.063068ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXXX.X.] duration: 637.925149ms FAIL (2025-08-07 23:18:07 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 637.946097ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXXX.X] duration: 30.001161408s FAIL (2025-08-07 23:18:06 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.00117864s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.002521228s FAIL (2025-08-07 23:17:57 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002531045s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 15.455679ms target: https://us1.storj.io/login FAIL (2025-08-07 23:17:54 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 15.478161ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................................................XXXXXX.X] duration: 30.000879404s FAIL (2025-08-07 23:17:51 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.00089845s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 30.001062664s FAIL (2025-08-07 23:17:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001083473s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [....................................................XXXXX.XX] duration: 30.000515064s FAIL (2025-08-07 23:17:42 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:04.749315126 (6h47m18.336371367s) [upload=6964]: Starting upload Aug 7 23:14:04.749394674 (6h47m18.336450915s) [upload=6964]: Done starting upload Aug 7 23:14:04.749474242 (6h47m18.336530484s) [upload=6964]: Uploading segments... Aug 7 23:14:04.749532461 (6h47m18.336588702s) [upload=6964]: Got next segment. Inline: true Aug 7 23:14:04.749538312 (6h47m18.336594553s) [upload=6964]: Waiting for error group managing segments... Aug 7 23:14:04.749584738 (6h47m18.33664098s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:04.749590058 (6h47m18.3366463s) [upload=6964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:04.749593755 (6h47m18.336649997s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:34.737068324 (6h47m48.324124536s) [upload=6964]: Done uploading segments... Aug 7 23:14:34.737089104 (6h47m48.324145314s) [upload=6964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.277141003 (6h47m50.864197244s) [upload=6974]: Starting upload Aug 7 23:14:37.277320197 (6h47m50.864376439s) [upload=6974]: Done starting upload Aug 7 23:14:37.277421095 (6h47m50.864477337s) [upload=6974]: Uploading segments... Aug 7 23:14:37.277519849 (6h47m50.864576091s) [upload=6974]: Got next segment. Inline: true Aug 7 23:14:37.277527635 (6h47m50.864583875s) [upload=6974]: Waiting for error group managing segments... Aug 7 23:14:37.277578299 (6h47m50.86463454s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.277584181 (6h47m50.864640421s) [upload=6974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.277587807 (6h47m50.864644049s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.263997045 (6h48m20.851053287s) [upload=6974]: Done uploading segments... Aug 7 23:15:07.264036289 (6h48m20.85109253s) [upload=6974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:40.767391975 (6h48m54.354448226s) [upload=6993]: Starting upload Aug 7 23:15:40.767481722 (6h48m54.354537963s) [upload=6993]: Done starting upload Aug 7 23:15:40.767544320 (6h48m54.35460057s) [upload=6993]: Uploading segments... Aug 7 23:15:40.767585375 (6h48m54.354641617s) [upload=6993]: Got next segment. Inline: true Aug 7 23:15:40.767591397 (6h48m54.354647638s) [upload=6993]: Waiting for error group managing segments... Aug 7 23:15:40.767637002 (6h48m54.354693244s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:40.767642423 (6h48m54.354698664s) [upload=6993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:40.767646530 (6h48m54.354702782s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:10.754586120 (6h49m24.341642341s) [upload=6993]: Done uploading segments... Aug 7 23:16:10.754601419 (6h49m24.34165763s) [upload=6993]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:26.719060301 (6h49m40.306116542s) [upload=7005]: Starting upload Aug 7 23:16:26.719141191 (6h49m40.306197433s) [upload=7005]: Done starting upload Aug 7 23:16:26.719210051 (6h49m40.306266292s) [upload=7005]: Uploading segments... Aug 7 23:16:26.719285351 (6h49m40.306341592s) [upload=7005]: Got next segment. Inline: true Aug 7 23:16:26.719292685 (6h49m40.306348926s) [upload=7005]: Waiting for error group managing segments... Aug 7 23:16:26.719340825 (6h49m40.306397066s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:26.719347277 (6h49m40.306403518s) [upload=7005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:26.719351064 (6h49m40.306407305s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:26.802515608 (6h49m40.389571849s) [upload=7005]: Done uploading segments... Aug 7 23:16:26.802535986 (6h49m40.389592227s) [upload=7005]: Upload finished. err: <nil> Aug 7 23:16:42.191377275 (6h49m55.778433516s) [upload=7011]: Starting upload Aug 7 23:16:42.191466421 (6h49m55.778522662s) [upload=7011]: Done starting upload Aug 7 23:16:42.191531582 (6h49m55.778587803s) [upload=7011]: Uploading segments... Aug 7 23:16:42.191607003 (6h49m55.778663224s) [upload=7011]: Got next segment. Inline: true Aug 7 23:16:42.191612414 (6h49m55.778668635s) [upload=7011]: Waiting for error group managing segments... Aug 7 23:16:42.191651346 (6h49m55.778707556s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:42.191656575 (6h49m55.778712787s) [upload=7011]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:42.191659481 (6h49m55.778715702s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.173345160 (6h50m25.760401401s) [upload=7011]: Done uploading segments... Aug 7 23:17:12.173364576 (6h50m25.760420816s) [upload=7011]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:12.261509015 (6h50m25.848565257s) [upload=7016]: Starting upload Aug 7 23:17:12.261597881 (6h50m25.848654122s) [upload=7016]: Done starting upload Aug 7 23:17:12.261665168 (6h50m25.848721419s) [upload=7016]: Uploading segments... Aug 7 23:17:12.261708137 (6h50m25.848764388s) [upload=7016]: Got next segment. Inline: true Aug 7 23:17:12.261713487 (6h50m25.848769728s) [upload=7016]: Waiting for error group managing segments... Aug 7 23:17:12.261755746 (6h50m25.848811988s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:12.261761196 (6h50m25.848817448s) [upload=7016]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:12.261764593 (6h50m25.848820834s) [upload=7016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:42.241895867 (6h50m55.828952078s) [upload=7016]: Done uploading segments... Aug 7 23:17:42.241913931 (6h50m55.828970141s) [upload=7016]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.00052903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 3.419504667s FAIL (2025-08-07 23:17:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277088798 (6h24m59.86414501s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277143912 (6h24m59.864200133s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277381786 (6h24m59.864437997s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277484318 (6h24m59.864540529s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277648944 (6h24m59.864705156s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277799085 (6h24m59.864855306s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.277909561 (6h24m59.864965772s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278182451 (6h24m59.865238672s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.278693113 (6h24m59.865749325s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281412349 (6h24m59.868468561s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281693725 (6h24m59.868749935s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.281992942 (6h24m59.869049154s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282287733 (6h24m59.869343954s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282546676 (6h24m59.869602897s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.282788578 (6h24m59.869844789s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Uploading piece... Aug 7 22:51:46.392536836 (6h24m59.979593076s) [upload=6559]: Next returned nil segment Aug 7 22:51:46.392553217 (6h24m59.979609467s) [upload=6559]: Waiting for error group managing segments... Aug 7 22:51:46.405547863 (6h24m59.992604074s) [noise=true , address=104.36.82.44:4896 , piece_id=4DC653OK7I4V73HIS5WG4VGJJB5YTOYXEQD2XNF2TZTD4CNVCS2A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407390732 (6h24m59.994446944s) [noise=true , address=104.36.82.16:4312 , piece_id=LN5J77ILTLNC5M3ILR5D5IIZWZ2W53WGP3NOH3DQGHNPONS2DTKA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.407641742 (6h24m59.994697992s) [noise=true , address=104.36.82.60:4337 , piece_id=6BYZFHJU4TOXGDUAMT4U2YENDSM3MTSVSDM2RVQVAK4VGS546UHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.414905489 (6h25m0.001961701s) [noise=true , address=104.36.82.37:4684 , piece_id=K7F72MNS2DPVT2YYXH55M2OZMNZX27JPDY7QHAQ6CILYCGQHKBHA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.416918417 (6h25m0.003974658s) [noise=true , address=104.36.82.43:4859 , piece_id=4KS6GFAXXBZZNZGJVQ3CX3YB5C6Y4OXBNMFJF2TK6GG7R3NNQ6TA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.418620614 (6h25m0.005676816s) [noise=true , address=104.36.82.61:4377 , piece_id=K4XIYP3PF6P3C7E54MCRTCILFE4SHZA4GAOIOQQRPR7KBSG5FDQA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.422324949 (6h25m0.00938116s) [noise=true , address=104.36.82.61:4398 , piece_id=E6BQEFYL5DIGMTWEGLH5YWZILDR7CXKYXCBZX5EAIGYJSRZ65GYQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.429110666 (6h25m0.016166907s) [noise=true , address=104.36.82.20:4455 , piece_id=BSLZU22ORRNH2LQZB2265XBJXKOI6JXFTSPTMMJ2UO2TXDJPDBCQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.431058481 (6h25m0.018114692s) [noise=true , address=104.36.82.62:4430 , piece_id=3R65ONSSTD6CANLJPLDPIFWN4XTH543NMSTJN5NE6GVEGQDSLQZA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.438518326 (6h25m0.025574537s) [noise=true , address=104.36.82.3:4796 , piece_id=RZDHQN3NYJ4Z7NNGU2WIOKAQUNVATUQMPD62YPBSAD7ZHKZKZCFQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.442266142 (6h25m0.029322354s) [noise=true , address=104.36.82.42:4812 , piece_id=ZUUJMTP5FVRZNYJDL4TCT6HH4EEEJSAREPJLZKE6T27CHYFKKJNA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496107844 (6h25m0.083164085s) [noise=true , address=104.36.82.35:4611 , piece_id=6ZG6W7PFEJWMFRVSDUUFPZFJWV6OKBJCPF3AS5EILT4MAEKQGTFA, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496492793 (6h25m0.083549034s) [noise=true , address=104.36.82.35:4598 , piece_id=UG2NRLNHD7O4K45FABP6P6NFF5AMMPD6K3IXOQT2H2N4VLI22N3A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496924579 (6h25m0.08398082s) [noise=true , address=104.36.82.46:4498 , piece_id=VIZMSHUDVCSGTZTFJZLKG6IU3RSBBBOBAHSCYAIEHF7EOOT3JFQQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: <nil> Aug 7 22:51:46.496949225 (6h25m0.084005476s) [seg_pos={0 0}, upload=6559]: Segment reached optimal threshold of 14 pieces. Aug 7 22:51:46.497368117 (6h25m0.084424357s) [noise=true , address=104.36.82.22:4537 , piece_id=22YN6FQ6GYOZGZZYXFNGEZ2GBWPX5RWIMMZPTYBIZMFX3D2XYTRQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y): protocol: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34872->104.36.82.22:4537: use of closed network connection Aug 7 22:51:46.497437055 (6h25m0.084493296s) [noise=true , address=104.36.82.59:4312 , piece_id=PVYVQDJDQST55KYNUL6T6I4TGPHBHEUAYBUYBUXJRESR7NZNY62A, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: read tcp 5.161.89.252:51322->104.36.82.59:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497509942 (6h25m0.084566182s) [noise=true , address=193.149.164.143:4697 , piece_id=MAE5V6WOR57X6C6HCP2WF3BWMQXLVPOTQTCIHDXFIZF45JXWZVPQ, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:46418->193.149.164.143:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:51:46.497555185 (6h25m0.084611437s) [noise=true , address=104.36.82.37:4680 , piece_id=CQFJPYOTEVSWBP7KO5EUSLGLW6CN7IEIQJQTXX7U2MS4V2PLVI4Q, seg_pos={0 0}, upload=6559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3): protocol: read tcp 5.161.89.252:37056->104.36.82.37:4680: use of closed network connection; conn closed: op Aug 7 22:51:46.497588538 (6h25m0.084644779s) [seg_pos={0 0}, upload=6559]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:51:46.497699846 (6h25m0.084756096s) [upload=6559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:51:46.497706178 (6h25m0.084762419s) [upload=6559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:51:46.589159026 (6h25m0.176215257s) [upload=6559]: Done uploading segments... Aug 7 22:51:46.589175717 (6h25m0.176231948s) [upload=6559]: Upload finished. err: <nil> Aug 7 22:52:02.401407012 (6h25m15.988463222s) [upload=6564]: Starting upload Aug 7 22:52:02.401455062 (6h25m15.988511283s) [upload=6564]: Done starting upload Aug 7 22:52:02.401530021 (6h25m15.988586232s) [upload=6564]: Uploading segments... Aug 7 22:52:02.401667688 (6h25m15.988723899s) [upload=6564]: Got next segment. Inline: false Aug 7 22:52:02.401691162 (6h25m15.988747383s) [upload=6564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:52:02.401694689 (6h25m15.9887509s) [upload=6564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:52:02.403805909 (6h25m15.990862131s) [upload=6564]: Next segment err: metaclient: Too Many Requests Aug 7 22:52:02.403815247 (6h25m15.990871468s) [upload=6564]: Done uploading segments... Aug 7 22:52:02.403823623 (6h25m15.990879834s) [upload=6564]: Upload finished. err: metaclient: Too Many Requests Duration: 3.419528672s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 8.427989039s FAIL (2025-08-07 23:17:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 8.428015939s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 13.981929ms target: https://us1.storj.io/login FAIL (2025-08-07 23:17:28 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 14.001825ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................X....XXX.] duration: 320.179434ms FAIL (2025-08-07 23:17:27 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 778582181 (6h47m34.365638402s) [noise=true , address=104.36.82.13:4187 , piece_id=7KPLHYJ6UUT3IHDTWCTBK4SZMPBMUCUFKWUKSIFTGWWQRB4NKSMQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.778802352 (6h47m34.365858573s) [noise=true , address=77.90.31.13:4628 , piece_id=XYY3WIIBUQ77VTGLP6IHAR42YHAALLHZWKP5WJWS5CY6KQGL47QQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779022393 (6h47m34.366078604s) [noise=true , address=45.91.116.10:4083 , piece_id=HWSKVUEIDBPUBNKM4WK23WOCXKNZ4WW5C7H4VWB2JPL6G7SU4PKA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779118041 (6h47m34.366174282s) [noise=true , address=104.36.82.14:4223 , piece_id=YRPBBRPJCCEGVU5Y5INAROPDZBCAPC65C3JPEYKOXMFGYYIFM3OA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779253655 (6h47m34.366309876s) [noise=true , address=77.90.31.15:4698 , piece_id=RTDRKWAMYFPTYYV342PJTHHWY2I7C7SIGMOKXHBTKIBTCSWR7ULQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779477572 (6h47m34.366533784s) [noise=true , address=104.36.82.16:4294 , piece_id=PDZEAAPWCP6CPWAVIHET4WIKNBZFP5CUO33DN7WUPGB6E7BK3B2Q, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779672236 (6h47m34.366728477s) [noise=true , address=77.90.31.19:4843 , piece_id=FER3W3CCPEAKOW43E4Z6SQAXCFOYBSUSKQE6F7LBLRF47BBXGQKQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.779697914 (6h47m34.366754135s) [noise=true , address=104.36.82.13:4185 , piece_id=6G45IZLNCGMIFCKO6ZFJIATGUODYG7I7LXCLT5KOBSLLJVUJBFJA, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.780260564 (6h47m34.367316805s) [noise=true , address=45.91.116.11:4117 , piece_id=DXVD6G72JWWPOQLEW5AFFOGPHVC56MT457KS6LMI66MTW4FTQLFQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.780779803 (6h47m34.367836044s) [noise=true , address=104.36.82.12:4151 , piece_id=RI53GSHPFHV7ARLXS5ROYUL32XCGQSIRBIKSVTKWM7QFTD5EME3Q, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.781487684 (6h47m34.368543936s) [noise=true , address=77.90.31.12:4592 , piece_id=UCU6L36WETOBSA3ENUBLFXKHHHCPI3BA27J22VZBNUD7DPBMHXDQ, seg_pos={0 0}, upload=6969]: Uploading piece... Aug 7 23:14:20.858608298 (6h47m34.445664509s) [upload=6969]: Next returned nil segment Aug 7 23:14:20.858630338 (6h47m34.445686559s) [upload=6969]: Waiting for error group managing segments... Aug 7 23:14:20.867703544 (6h47m34.454759765s) [noise=true , address=45.91.116.6:4973 , piece_id=JNT2NU7DK7LVLBF4CK3Q3H2L5VNAEYZTFGEAAEYTRSN5QNLCMA2Q, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.869347483 (6h47m34.456403704s) [noise=true , address=77.90.31.13:4628 , piece_id=XYY3WIIBUQ77VTGLP6IHAR42YHAALLHZWKP5WJWS5CY6KQGL47QQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.869615523 (6h47m34.456671734s) [noise=true , address=77.90.31.16:4733 , piece_id=5WOTQCOZFOGGM7KVYXQM5F3KHKKY62BVLUOFVNL43RG6B46N6LSA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.870768575 (6h47m34.457824796s) [noise=true , address=45.91.116.4:4939 , piece_id=AJUDS6MJG526UTQOND235L6MKAD3AIKR3J6MREXIZ5OMVFCOH5ZA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.872453060 (6h47m34.459509281s) [noise=true , address=77.90.31.15:4698 , piece_id=RTDRKWAMYFPTYYV342PJTHHWY2I7C7SIGMOKXHBTKIBTCSWR7ULQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.873530410 (6h47m34.460586621s) [noise=true , address=77.90.31.18:4807 , piece_id=M2V6MVI7EZDBIKKIP3N7YARK2YYOPDJYX2SDPIIMNUFLCGO34RDA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.874615627 (6h47m34.461671838s) [noise=true , address=45.91.116.10:4083 , piece_id=HWSKVUEIDBPUBNKM4WK23WOCXKNZ4WW5C7H4VWB2JPL6G7SU4PKA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.875179007 (6h47m34.462235218s) [noise=true , address=77.90.31.12:4592 , piece_id=UCU6L36WETOBSA3ENUBLFXKHHHCPI3BA27J22VZBNUD7DPBMHXDQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.880399252 (6h47m34.467455473s) [noise=true , address=45.91.116.11:4117 , piece_id=DXVD6G72JWWPOQLEW5AFFOGPHVC56MT457KS6LMI66MTW4FTQLFQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.883816901 (6h47m34.470873122s) [noise=true , address=104.36.82.14:4223 , piece_id=YRPBBRPJCCEGVU5Y5INAROPDZBCAPC65C3JPEYKOXMFGYYIFM3OA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.886328790 (6h47m34.473385001s) [noise=true , address=104.36.82.13:4187 , piece_id=7KPLHYJ6UUT3IHDTWCTBK4SZMPBMUCUFKWUKSIFTGWWQRB4NKSMQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.889471356 (6h47m34.476527577s) [noise=true , address=77.90.31.19:4843 , piece_id=FER3W3CCPEAKOW43E4Z6SQAXCFOYBSUSKQE6F7LBLRF47BBXGQKQ, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.893860889 (6h47m34.48091711s) [noise=true , address=104.36.82.13:4185 , piece_id=6G45IZLNCGMIFCKO6ZFJIATGUODYG7I7LXCLT5KOBSLLJVUJBFJA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: <nil> Aug 7 23:14:20.893882650 (6h47m34.48093886s) [seg_pos={0 0}, upload=6969]: Segment reached optimal threshold of 13 pieces. Aug 7 23:14:20.894050533 (6h47m34.481106754s) [noise=true , address=77.90.31.21:4877 , piece_id=CZDLZNZECZ6QDL5L4DKL6B7PVPEQFESMN7ZNMRKD6ZNYLZWDVUSA, seg_pos={0 0}, upload=6969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Aug 7 23:14:20.894105426 (6h47m34.481161656s) [noise=true , address=104.36.82.12:4151 , piece_id=RI53GSHPFHV7ARLXS5ROYUL32XCGQSIRBIKSVTKWM7QFTD5EME3Q, seg_pos={0 0}, upload=6969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 5.161.89.252:8836->104.36.82.12:4151: use of closed network connection; write tcp 5.161.89.252:8836->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:8836->104.36.82.12:4151: use of closed network connection Aug 7 23:14:20.894188311 (6h47m34.481244532s) [noise=true , address=104.36.82.16:4294 , piece_id=PDZEAAPWCP6CPWAVIHET4WIKNBZFP5CUO33DN7WUPGB6E7BK3B2Q, seg_pos={0 0}, upload=6969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: read tcp 5.161.89.252:36452->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:14:20.894198529 (6h47m34.48125475s) [seg_pos={0 0}, upload=6969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:14:20.894279671 (6h47m34.481335882s) [upload=6969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:14:20.894283668 (6h47m34.481339889s) [upload=6969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:20.936620403 (6h47m34.523676654s) [upload=6969]: Done uploading segments... Aug 7 23:14:20.936636754 (6h47m34.523692975s) [upload=6969]: Upload finished. err: <nil> Aug 7 23:14:44.515675412 (6h47m58.102731643s) [upload=6976]: Starting upload Aug 7 23:14:44.515752076 (6h47m58.102808317s) [upload=6976]: Done starting upload Aug 7 23:14:44.515823649 (6h47m58.10287991s) [upload=6976]: Uploading segments... Aug 7 23:14:44.516164195 (6h47m58.103220436s) [upload=6976]: Got next segment. Inline: false Aug 7 23:14:44.516248702 (6h47m58.103304944s) [upload=6976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:44.516256998 (6h47m58.103313239s) [upload=6976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:14.501722064 (6h48m28.088778285s) [upload=6976]: Next segment err: context deadline exceeded Aug 7 23:15:14.501867856 (6h48m28.088924078s) [upload=6976]: Done uploading segments... Aug 7 23:15:14.501885520 (6h48m28.088941731s) [upload=6976]: Upload finished. err: context deadline exceeded Aug 7 23:15:50.351511032 (6h49m3.938567273s) [upload=6996]: Starting upload Aug 7 23:15:50.351596221 (6h49m3.938652472s) [upload=6996]: Done starting upload Aug 7 23:15:50.351682441 (6h49m3.938738683s) [upload=6996]: Uploading segments... Aug 7 23:15:50.351994003 (6h49m3.939050244s) [upload=6996]: Got next segment. Inline: false Aug 7 23:15:50.352036763 (6h49m3.939093004s) [upload=6996]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:50.352042523 (6h49m3.939098765s) [upload=6996]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:20.337163793 (6h49m33.924220044s) [upload=6996]: Next segment err: context deadline exceeded Aug 7 23:16:20.337425733 (6h49m33.924481983s) [upload=6996]: Done uploading segments... Aug 7 23:16:20.337475926 (6h49m33.924532177s) [upload=6996]: Upload finished. err: context deadline exceeded Aug 7 23:16:20.420371668 (6h49m34.007427909s) [upload=7003]: Starting upload Aug 7 23:16:20.420456566 (6h49m34.007512807s) [upload=7003]: Done starting upload Aug 7 23:16:20.420524824 (6h49m34.007581066s) [upload=7003]: Uploading segments... Aug 7 23:16:20.420851844 (6h49m34.007908086s) [upload=7003]: Got next segment. Inline: false Aug 7 23:16:20.420905824 (6h49m34.007962066s) [upload=7003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:20.420912007 (6h49m34.007968247s) [upload=7003]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:50.407120057 (6h50m3.994176297s) [upload=7003]: Next segment err: context deadline exceeded Aug 7 23:16:50.407194757 (6h50m3.994250997s) [upload=7003]: Done uploading segments... Aug 7 23:16:50.407241563 (6h50m3.994297814s) [upload=7003]: Upload finished. err: context deadline exceeded Duration: 320.207137ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.000376835s FAIL (2025-08-07 23:17:23 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 941171742 (6h42m10.528227963s) [upload=6873]: Done uploading segments... Aug 7 23:08:56.941188142 (6h42m10.528244354s) [upload=6873]: Upload finished. err: <nil> Aug 7 23:09:42.944491406 (6h42m56.531547628s) [upload=6888]: Starting upload Aug 7 23:09:42.944574772 (6h42m56.531630993s) [upload=6888]: Done starting upload Aug 7 23:09:42.944630365 (6h42m56.531686576s) [upload=6888]: Uploading segments... Aug 7 23:09:42.944668176 (6h42m56.531724397s) [upload=6888]: Got next segment. Inline: true Aug 7 23:09:42.944673766 (6h42m56.531729977s) [upload=6888]: Waiting for error group managing segments... Aug 7 23:09:42.944715465 (6h42m56.531771686s) [upload=6888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:42.944720784 (6h42m56.531777005s) [upload=6888]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:42.944724171 (6h42m56.531780392s) [upload=6888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:43.008585323 (6h42m56.595641534s) [upload=6888]: Done uploading segments... Aug 7 23:09:43.008607604 (6h42m56.595663825s) [upload=6888]: Upload finished. err: <nil> Aug 7 23:10:07.090790160 (6h43m20.677846371s) [upload=6895]: Starting upload Aug 7 23:10:07.090866883 (6h43m20.677923094s) [upload=6895]: Done starting upload Aug 7 23:10:07.090916526 (6h43m20.677972737s) [upload=6895]: Uploading segments... Aug 7 23:10:07.091175900 (6h43m20.678232142s) [upload=6895]: Got next segment. Inline: true Aug 7 23:10:07.091186920 (6h43m20.678243141s) [upload=6895]: Waiting for error group managing segments... Aug 7 23:10:07.091251632 (6h43m20.678307853s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:07.091257803 (6h43m20.678314014s) [upload=6895]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:07.091262211 (6h43m20.678318423s) [upload=6895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:07.150936870 (6h43m20.737993091s) [upload=6895]: Done uploading segments... Aug 7 23:10:07.150955715 (6h43m20.738011936s) [upload=6895]: Upload finished. err: <nil> Aug 7 23:10:35.935324537 (6h43m49.522380777s) [upload=6905]: Starting upload Aug 7 23:10:35.935406759 (6h43m49.522463001s) [upload=6905]: Done starting upload Aug 7 23:10:35.935481139 (6h43m49.52253739s) [upload=6905]: Uploading segments... Aug 7 23:10:35.935518358 (6h43m49.522574599s) [upload=6905]: Got next segment. Inline: true Aug 7 23:10:35.935523989 (6h43m49.522580229s) [upload=6905]: Waiting for error group managing segments... Aug 7 23:10:35.935567379 (6h43m49.522623621s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:35.935572860 (6h43m49.522629101s) [upload=6905]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:35.935576386 (6h43m49.522632628s) [upload=6905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.018368185 (6h43m49.605424426s) [upload=6905]: Done uploading segments... Aug 7 23:10:36.018385587 (6h43m49.605441827s) [upload=6905]: Upload finished. err: <nil> Aug 7 23:10:57.038963659 (6h44m10.62601988s) [upload=6913]: Starting upload Aug 7 23:10:57.039017149 (6h44m10.62607337s) [upload=6913]: Done starting upload Aug 7 23:10:57.039065159 (6h44m10.62612137s) [upload=6913]: Uploading segments... Aug 7 23:10:57.039093831 (6h44m10.626150042s) [upload=6913]: Got next segment. Inline: true Aug 7 23:10:57.039097308 (6h44m10.626153519s) [upload=6913]: Waiting for error group managing segments... Aug 7 23:10:57.039142663 (6h44m10.626198874s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:57.039146320 (6h44m10.626202531s) [upload=6913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:57.039148564 (6h44m10.626204785s) [upload=6913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:57.092133855 (6h44m10.679190086s) [upload=6913]: Done uploading segments... Aug 7 23:10:57.092152090 (6h44m10.67920833s) [upload=6913]: Upload finished. err: <nil> Aug 7 23:11:47.690012005 (6h45m1.277068256s) [upload=6926]: Starting upload Aug 7 23:11:47.690127641 (6h45m1.277183882s) [upload=6926]: Done starting upload Aug 7 23:11:47.690277249 (6h45m1.27733349s) [upload=6926]: Uploading segments... Aug 7 23:11:47.690316773 (6h45m1.277373025s) [upload=6926]: Got next segment. Inline: true Aug 7 23:11:47.690360005 (6h45m1.277416245s) [upload=6926]: Waiting for error group managing segments... Aug 7 23:11:47.690405779 (6h45m1.277462031s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:47.690411880 (6h45m1.277468122s) [upload=6926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:47.690415508 (6h45m1.277471748s) [upload=6926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:47.740982296 (6h45m1.328038537s) [upload=6926]: Done uploading segments... Aug 7 23:11:47.740995380 (6h45m1.328051622s) [upload=6926]: Upload finished. err: <nil> Aug 7 23:12:15.388374517 (6h45m28.975430749s) [upload=6935]: Starting upload Aug 7 23:12:15.388464206 (6h45m28.975520437s) [upload=6935]: Done starting upload Aug 7 23:12:15.388550818 (6h45m28.975607039s) [upload=6935]: Uploading segments... Aug 7 23:12:15.388651916 (6h45m28.975708127s) [upload=6935]: Got next segment. Inline: true Aug 7 23:12:15.388660222 (6h45m28.975716443s) [upload=6935]: Waiting for error group managing segments... Aug 7 23:12:15.388716747 (6h45m28.975772968s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:15.388723729 (6h45m28.97577995s) [upload=6935]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:15.388727838 (6h45m28.975784059s) [upload=6935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:15.375897715 (6h46m28.962953947s) [upload=6935]: Done uploading segments... Aug 7 23:13:15.375942599 (6h46m28.962998841s) [upload=6935]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:15.452799710 (6h46m29.039855931s) [upload=6951]: Starting upload Aug 7 23:13:15.452856376 (6h46m29.039912597s) [upload=6951]: Done starting upload Aug 7 23:13:15.452901079 (6h46m29.0399573s) [upload=6951]: Uploading segments... Aug 7 23:13:15.452938739 (6h46m29.03999495s) [upload=6951]: Got next segment. Inline: true Aug 7 23:13:15.452943217 (6h46m29.039999439s) [upload=6951]: Waiting for error group managing segments... Aug 7 23:13:15.452974977 (6h46m29.040031188s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:15.452979305 (6h46m29.040035526s) [upload=6951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:15.452982110 (6h46m29.040038331s) [upload=6951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:15.442587501 (6h47m29.029643742s) [upload=6951]: Done uploading segments... Aug 7 23:14:15.442608670 (6h47m29.029664922s) [upload=6951]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:15.516163727 (6h47m29.103219978s) [upload=6968]: Starting upload Aug 7 23:14:15.516314939 (6h47m29.10337124s) [upload=6968]: Done starting upload Aug 7 23:14:15.516411479 (6h47m29.103467721s) [upload=6968]: Uploading segments... Aug 7 23:14:15.516477332 (6h47m29.103533573s) [upload=6968]: Got next segment. Inline: true Aug 7 23:14:15.516487672 (6h47m29.103543913s) [upload=6968]: Waiting for error group managing segments... Aug 7 23:14:15.516556882 (6h47m29.103613162s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:15.516567169 (6h47m29.103623411s) [upload=6968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:15.516573491 (6h47m29.103629733s) [upload=6968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:15.501845849 (6h48m29.08890207s) [upload=6968]: Done uploading segments... Aug 7 23:15:15.501865025 (6h48m29.088921247s) [upload=6968]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:15.591828711 (6h48m29.178884922s) [upload=6986]: Starting upload Aug 7 23:15:15.591883403 (6h48m29.178939613s) [upload=6986]: Done starting upload Aug 7 23:15:15.591959133 (6h48m29.179015344s) [upload=6986]: Uploading segments... Aug 7 23:15:15.592001994 (6h48m29.179058205s) [upload=6986]: Got next segment. Inline: true Aug 7 23:15:15.592022743 (6h48m29.179078964s) [upload=6986]: Waiting for error group managing segments... Aug 7 23:15:15.592060944 (6h48m29.179117156s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:15.592065653 (6h48m29.179121874s) [upload=6986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:15.592067957 (6h48m29.179124178s) [upload=6986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:15.578451341 (6h49m29.165507562s) [upload=6986]: Done uploading segments... Aug 7 23:16:15.578469606 (6h49m29.165525816s) [upload=6986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:23.278937820 (6h49m36.865994061s) [upload=7004]: Starting upload Aug 7 23:16:23.279020784 (6h49m36.866077025s) [upload=7004]: Done starting upload Aug 7 23:16:23.279084963 (6h49m36.866141205s) [upload=7004]: Uploading segments... Aug 7 23:16:23.279128285 (6h49m36.866184526s) [upload=7004]: Got next segment. Inline: true Aug 7 23:16:23.279135057 (6h49m36.866191299s) [upload=7004]: Waiting for error group managing segments... Aug 7 23:16:23.279179039 (6h49m36.86623528s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:23.279184509 (6h49m36.866240751s) [upload=7004]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:23.279188327 (6h49m36.866244558s) [upload=7004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:23.264696428 (6h50m36.851752649s) [upload=7004]: Done uploading segments... Aug 7 23:17:23.264712168 (6h50m36.851768379s) [upload=7004]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000396203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XXXXX.X] duration: 30.001174006s FAIL (2025-08-07 23:17:12 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [upload=6901]: Done uploading segments... Aug 7 23:10:19.553720001 (6h43m33.140776211s) [upload=6901]: Upload finished. err: <nil> Aug 7 23:10:36.579109754 (6h43m50.166165975s) [upload=6906]: Starting upload Aug 7 23:10:36.579181208 (6h43m50.166237419s) [upload=6906]: Done starting upload Aug 7 23:10:36.579283839 (6h43m50.166340051s) [upload=6906]: Uploading segments... Aug 7 23:10:36.579316520 (6h43m50.166372741s) [upload=6906]: Got next segment. Inline: true Aug 7 23:10:36.579320318 (6h43m50.166376529s) [upload=6906]: Waiting for error group managing segments... Aug 7 23:10:36.579352938 (6h43m50.166409159s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:36.579356855 (6h43m50.166413076s) [upload=6906]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:36.579358979 (6h43m50.16641519s) [upload=6906]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:36.629114792 (6h43m50.216171034s) [upload=6906]: Done uploading segments... Aug 7 23:10:36.629132566 (6h43m50.216188797s) [upload=6906]: Upload finished. err: <nil> Aug 7 23:11:05.687805040 (6h44m19.274861281s) [upload=6916]: Starting upload Aug 7 23:11:05.687992249 (6h44m19.27504857s) [upload=6916]: Done starting upload Aug 7 23:11:05.688099038 (6h44m19.275155279s) [upload=6916]: Uploading segments... Aug 7 23:11:05.688157528 (6h44m19.275213799s) [upload=6916]: Got next segment. Inline: true Aug 7 23:11:05.688166765 (6h44m19.275223017s) [upload=6916]: Waiting for error group managing segments... Aug 7 23:11:05.688260139 (6h44m19.27531638s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:05.688269687 (6h44m19.275325969s) [upload=6916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:05.688275919 (6h44m19.27533216s) [upload=6916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:05.740655239 (6h44m19.327711451s) [upload=6916]: Done uploading segments... Aug 7 23:11:05.740669697 (6h44m19.327725918s) [upload=6916]: Upload finished. err: <nil> Aug 7 23:12:08.089702100 (6h45m21.67675834s) [upload=6934]: Starting upload Aug 7 23:12:08.089804582 (6h45m21.676860822s) [upload=6934]: Done starting upload Aug 7 23:12:08.089877487 (6h45m21.676933729s) [upload=6934]: Uploading segments... Aug 7 23:12:08.089928542 (6h45m21.676984783s) [upload=6934]: Got next segment. Inline: true Aug 7 23:12:08.089934242 (6h45m21.676990484s) [upload=6934]: Waiting for error group managing segments... Aug 7 23:12:08.089980470 (6h45m21.67703671s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:08.089986311 (6h45m21.677042551s) [upload=6934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:08.089990338 (6h45m21.677046579s) [upload=6934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:08.195718921 (6h45m21.782775162s) [upload=6934]: Done uploading segments... Aug 7 23:12:08.195734549 (6h45m21.78279079s) [upload=6934]: Upload finished. err: <nil> Aug 7 23:12:48.910499617 (6h46m2.497555858s) [upload=6943]: Starting upload Aug 7 23:12:48.910615834 (6h46m2.497672085s) [upload=6943]: Done starting upload Aug 7 23:12:48.910681648 (6h46m2.497737898s) [upload=6943]: Uploading segments... Aug 7 23:12:48.910722363 (6h46m2.497778604s) [upload=6943]: Got next segment. Inline: true Aug 7 23:12:48.910727974 (6h46m2.497784214s) [upload=6943]: Waiting for error group managing segments... Aug 7 23:12:48.910775202 (6h46m2.497831443s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:48.910780953 (6h46m2.497837204s) [upload=6943]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:48.910784499 (6h46m2.497840741s) [upload=6943]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.893357638 (6h46m32.480413889s) [upload=6943]: Done uploading segments... Aug 7 23:13:18.893388065 (6h46m32.480444306s) [upload=6943]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:18.972282939 (6h46m32.559339181s) [upload=6953]: Starting upload Aug 7 23:13:18.972362728 (6h46m32.55941897s) [upload=6953]: Done starting upload Aug 7 23:13:18.972440393 (6h46m32.559496634s) [upload=6953]: Uploading segments... Aug 7 23:13:18.972481941 (6h46m32.559538181s) [upload=6953]: Got next segment. Inline: true Aug 7 23:13:18.972487681 (6h46m32.559543923s) [upload=6953]: Waiting for error group managing segments... Aug 7 23:13:18.972531043 (6h46m32.559587284s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.972536602 (6h46m32.559592843s) [upload=6953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.972540159 (6h46m32.5595964s) [upload=6953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:48.959236568 (6h47m2.546292809s) [upload=6953]: Done uploading segments... Aug 7 23:13:48.959259700 (6h47m2.546315941s) [upload=6953]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:04.749315126 (6h47m18.336371367s) [upload=6964]: Starting upload Aug 7 23:14:04.749394674 (6h47m18.336450915s) [upload=6964]: Done starting upload Aug 7 23:14:04.749474242 (6h47m18.336530484s) [upload=6964]: Uploading segments... Aug 7 23:14:04.749532461 (6h47m18.336588702s) [upload=6964]: Got next segment. Inline: true Aug 7 23:14:04.749538312 (6h47m18.336594553s) [upload=6964]: Waiting for error group managing segments... Aug 7 23:14:04.749584738 (6h47m18.33664098s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:04.749590058 (6h47m18.3366463s) [upload=6964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:04.749593755 (6h47m18.336649997s) [upload=6964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:34.737068324 (6h47m48.324124536s) [upload=6964]: Done uploading segments... Aug 7 23:14:34.737089104 (6h47m48.324145314s) [upload=6964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.277141003 (6h47m50.864197244s) [upload=6974]: Starting upload Aug 7 23:14:37.277320197 (6h47m50.864376439s) [upload=6974]: Done starting upload Aug 7 23:14:37.277421095 (6h47m50.864477337s) [upload=6974]: Uploading segments... Aug 7 23:14:37.277519849 (6h47m50.864576091s) [upload=6974]: Got next segment. Inline: true Aug 7 23:14:37.277527635 (6h47m50.864583875s) [upload=6974]: Waiting for error group managing segments... Aug 7 23:14:37.277578299 (6h47m50.86463454s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.277584181 (6h47m50.864640421s) [upload=6974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.277587807 (6h47m50.864644049s) [upload=6974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.263997045 (6h48m20.851053287s) [upload=6974]: Done uploading segments... Aug 7 23:15:07.264036289 (6h48m20.85109253s) [upload=6974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:40.767391975 (6h48m54.354448226s) [upload=6993]: Starting upload Aug 7 23:15:40.767481722 (6h48m54.354537963s) [upload=6993]: Done starting upload Aug 7 23:15:40.767544320 (6h48m54.35460057s) [upload=6993]: Uploading segments... Aug 7 23:15:40.767585375 (6h48m54.354641617s) [upload=6993]: Got next segment. Inline: true Aug 7 23:15:40.767591397 (6h48m54.354647638s) [upload=6993]: Waiting for error group managing segments... Aug 7 23:15:40.767637002 (6h48m54.354693244s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:40.767642423 (6h48m54.354698664s) [upload=6993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:40.767646530 (6h48m54.354702782s) [upload=6993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:10.754586120 (6h49m24.341642341s) [upload=6993]: Done uploading segments... Aug 7 23:16:10.754601419 (6h49m24.34165763s) [upload=6993]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:26.719060301 (6h49m40.306116542s) [upload=7005]: Starting upload Aug 7 23:16:26.719141191 (6h49m40.306197433s) [upload=7005]: Done starting upload Aug 7 23:16:26.719210051 (6h49m40.306266292s) [upload=7005]: Uploading segments... Aug 7 23:16:26.719285351 (6h49m40.306341592s) [upload=7005]: Got next segment. Inline: true Aug 7 23:16:26.719292685 (6h49m40.306348926s) [upload=7005]: Waiting for error group managing segments... Aug 7 23:16:26.719340825 (6h49m40.306397066s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:26.719347277 (6h49m40.306403518s) [upload=7005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:26.719351064 (6h49m40.306407305s) [upload=7005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:26.802515608 (6h49m40.389571849s) [upload=7005]: Done uploading segments... Aug 7 23:16:26.802535986 (6h49m40.389592227s) [upload=7005]: Upload finished. err: <nil> Aug 7 23:16:42.191377275 (6h49m55.778433516s) [upload=7011]: Starting upload Aug 7 23:16:42.191466421 (6h49m55.778522662s) [upload=7011]: Done starting upload Aug 7 23:16:42.191531582 (6h49m55.778587803s) [upload=7011]: Uploading segments... Aug 7 23:16:42.191607003 (6h49m55.778663224s) [upload=7011]: Got next segment. Inline: true Aug 7 23:16:42.191612414 (6h49m55.778668635s) [upload=7011]: Waiting for error group managing segments... Aug 7 23:16:42.191651346 (6h49m55.778707556s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:42.191656575 (6h49m55.778712787s) [upload=7011]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:42.191659481 (6h49m55.778715702s) [upload=7011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.173345160 (6h50m25.760401401s) [upload=7011]: Done uploading segments... Aug 7 23:17:12.173364576 (6h50m25.760420816s) [upload=7011]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001199975s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 11.60381ms target: https://us1.storj.io/login FAIL (2025-08-07 23:17:07 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 11.619679ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................X.XX.X.] duration: 477.653219ms FAIL (2025-08-07 23:17:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 477.671063ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XXXXXX.] duration: 1.849439974s FAIL (2025-08-07 23:17:04 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.849463127s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 814.294899ms FAIL (2025-08-07 23:17:02 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 1:4407 , piece_id=OZ4NVFVFPH3DRK6SZZ5KNKSRH4CJ54PMVAU44IUZCLO5INBGMMCA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487434914 (6h47m1.074491205s) [noise=true , address=104.36.82.9:4026 , piece_id=PA5TEDXTLF2VUXS3ZKJQVN3XA5YOWYTRQ7JHVFJKR6R7FNRKLKQA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487589051 (6h47m1.074645282s) [noise=true , address=104.36.82.44:4877 , piece_id=2DMZUKAPQW3IPUDMQHEHHDY2AKBODJH2HXVLJQLODP65PJRWUU7A, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487895814 (6h47m1.074952045s) [noise=true , address=45.91.116.4:4959 , piece_id=L4UEVAK4XOTCYG7NJA5ZUSA6M5NVXSLCYWLGS2D4IJ72MBSMHDQQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488059319 (6h47m1.07511555s) [noise=true , address=156.146.43.225:4458 , piece_id=PSBWZ62GHSTTCQNYW5JCUJ3TNGISQTK7UZGBYMM6NJBEO7RNKBOA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488340534 (6h47m1.075396765s) [noise=true , address=104.36.82.8:4996 , piece_id=FL2UEVQWZ3LU7GJLG3DQEUANVU6OOF3TURY6PYRKV4YHV73JNTZA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488536069 (6h47m1.0755923s) [noise=true , address=104.36.82.60:4361 , piece_id=KRZZ7RVKA7OEKNSKRSFDBVFPCO3436LVER5ZYWX2DQGWV66D5WAQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488721475 (6h47m1.075777716s) [noise=true , address=193.149.164.143:4697 , piece_id=UA3RGUUMVUIJQ2MHCSNNAI6TXQQLYKHYT6D4XUWDJFOCEFRV7OEQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.489079183 (6h47m1.076135424s) [noise=true , address=104.36.82.5:4868 , piece_id=VKYGMYQJJATRZV45S4AKZOFDF7ZP66FJNHV7WZY6PATZKNB4F74Q, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.489506110 (6h47m1.07656235s) [noise=true , address=104.36.82.41:4786 , piece_id=CYINWGSZE6EFPKWO5AZKME44HKOPHA22Y2O252GOGJHNGFV2VMNA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.489945269 (6h47m1.077001511s) [noise=true , address=79.127.201.215:4356 , piece_id=6NUG6TPMJMVSOWQTR5WQUE5MDGEBZSNVECQNAUUD5PUN2473HOOA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.490429594 (6h47m1.077485834s) [noise=true , address=104.36.82.9:4040 , piece_id=AFBZHJHUTHJSYXYQQIQ6XTVIFBVO6G5LSA4VARM4JCKJPAXSQS4Q, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.490819892 (6h47m1.077876133s) [noise=true , address=104.36.82.54:4174 , piece_id=4YFSUNEDHTNLMMJOTPMEZBB325UR2VAONVTELFNX6O62UE3AUBUA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.574630371 (6h47m1.161686592s) [upload=6952]: Next returned nil segment Aug 7 23:13:47.574641062 (6h47m1.161697283s) [upload=6952]: Waiting for error group managing segments... Aug 7 23:13:47.587541340 (6h47m1.174597562s) [noise=true , address=104.36.82.41:4786 , piece_id=CYINWGSZE6EFPKWO5AZKME44HKOPHA22Y2O252GOGJHNGFV2VMNA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.592242405 (6h47m1.179298626s) [noise=true , address=104.36.82.39:4748 , piece_id=SJ2OPHKNEGN4YAHZAYB2RHLCNHUFFUQBA7WEQP6BWCBRYWDJIYVQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.599971292 (6h47m1.187027503s) [noise=true , address=104.36.82.9:4040 , piece_id=AFBZHJHUTHJSYXYQQIQ6XTVIFBVO6G5LSA4VARM4JCKJPAXSQS4Q, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.607959633 (6h47m1.195015854s) [noise=true , address=45.91.116.11:4133 , piece_id=ZY3OVRW2WOSIW535LHRPPNPCOE75B26VPK2S27Z6KRKWVDSREWFA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.611555756 (6h47m1.198611967s) [noise=true , address=104.36.82.58:4279 , piece_id=AH5MJW4W346CNJN6LPTKW3FNQV6SUWP5JC47HDH5SRU4D7OWI43Q, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.626811131 (6h47m1.213867352s) [noise=true , address=45.91.116.4:4959 , piece_id=L4UEVAK4XOTCYG7NJA5ZUSA6M5NVXSLCYWLGS2D4IJ72MBSMHDQQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.637742776 (6h47m1.224799007s) [noise=true , address=104.36.82.5:4868 , piece_id=VKYGMYQJJATRZV45S4AKZOFDF7ZP66FJNHV7WZY6PATZKNB4F74Q, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.639233228 (6h47m1.226289449s) [noise=true , address=104.36.82.60:4361 , piece_id=KRZZ7RVKA7OEKNSKRSFDBVFPCO3436LVER5ZYWX2DQGWV66D5WAQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.639680683 (6h47m1.226736904s) [noise=true , address=104.36.82.44:4877 , piece_id=2DMZUKAPQW3IPUDMQHEHHDY2AKBODJH2HXVLJQLODP65PJRWUU7A, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.646508387 (6h47m1.233564599s) [noise=true , address=104.36.82.54:4174 , piece_id=4YFSUNEDHTNLMMJOTPMEZBB325UR2VAONVTELFNX6O62UE3AUBUA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.669751322 (6h47m1.256807543s) [noise=true , address=104.36.82.8:4996 , piece_id=FL2UEVQWZ3LU7GJLG3DQEUANVU6OOF3TURY6PYRKV4YHV73JNTZA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.722599178 (6h47m1.309655429s) [noise=true , address=79.127.201.215:4359 , piece_id=ROGWPZSHIMRJHEF7EHHDFH2E4VZQBE5ESC6EJV623NMGE7PUGW2A, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.724255018 (6h47m1.31131124s) [noise=true , address=156.146.43.225:4458 , piece_id=PSBWZ62GHSTTCQNYW5JCUJ3TNGISQTK7UZGBYMM6NJBEO7RNKBOA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.740407329 (6h47m1.32746358s) [noise=true , address=79.127.201.215:4356 , piece_id=6NUG6TPMJMVSOWQTR5WQUE5MDGEBZSNVECQNAUUD5PUN2473HOOA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.740458694 (6h47m1.327514935s) [seg_pos={0 0}, upload=6952]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:47.740645193 (6h47m1.327701414s) [noise=true , address=104.36.82.9:4026 , piece_id=PA5TEDXTLF2VUXS3ZKJQVN3XA5YOWYTRQ7JHVFJKR6R7FNRKLKQA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1M7HJBLoxHkRHXbPuDiqwxdb4EAq5wktfJY5FygdoDdgy59rRq): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:47.740952506 (6h47m1.328008727s) [noise=true , address=64.49.13.169:4713 , piece_id=UOCWLHFGT2MI3K4IIDO3YIFJIXZGMCRWF55QSBRIKDZD5DSBQDVA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: read tcp 5.161.89.252:25140->64.49.13.169:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:47.741048936 (6h47m1.328105147s) [noise=true , address=104.36.82.61:4407 , piece_id=OZ4NVFVFPH3DRK6SZZ5KNKSRH4CJ54PMVAU44IUZCLO5INBGMMCA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122k7b6wa2JhSXMVAhF4qPKA48cWkh3DZRyWuwPEt7Vp3ciofPF): protocol: read tcp 5.161.89.252:36630->104.36.82.61:4407: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:47.741097086 (6h47m1.328153327s) [noise=true , address=193.149.164.143:4697 , piece_id=UA3RGUUMVUIJQ2MHCSNNAI6TXQQLYKHYT6D4XUWDJFOCEFRV7OEQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:22104->193.149.164.143:4697: use of closed network connection; conn closed: op Aug 7 23:13:47.741147290 (6h47m1.32820353s) [seg_pos={0 0}, upload=6952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:47.741286780 (6h47m1.328343021s) [upload=6952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:47.741297580 (6h47m1.328353832s) [upload=6952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:59.351303634 (6h48m12.938359875s) [upload=6952]: Done uploading segments... Aug 7 23:14:59.351331766 (6h48m12.938388007s) [upload=6952]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:59.433768531 (6h48m13.020824773s) [upload=6982]: Starting upload Aug 7 23:14:59.433862216 (6h48m13.020918457s) [upload=6982]: Done starting upload Aug 7 23:14:59.434026252 (6h48m13.021082504s) [upload=6982]: Uploading segments... Aug 7 23:14:59.434316203 (6h48m13.021372445s) [upload=6982]: Got next segment. Inline: false Aug 7 23:14:59.434383048 (6h48m13.02143929s) [upload=6982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.434389380 (6h48m13.021445622s) [upload=6982]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:59.416073113 (6h49m13.003129334s) [upload=6982]: Next segment err: context deadline exceeded Aug 7 23:15:59.416143385 (6h49m13.003199606s) [upload=6982]: Done uploading segments... Aug 7 23:15:59.416158513 (6h49m13.003214734s) [upload=6982]: Upload finished. err: context deadline exceeded Aug 7 23:15:59.510497796 (6h49m13.097554017s) [upload=6997]: Starting upload Aug 7 23:15:59.510549312 (6h49m13.097605533s) [upload=6997]: Done starting upload Aug 7 23:15:59.510663394 (6h49m13.097719606s) [upload=6997]: Uploading segments... Aug 7 23:15:59.510782647 (6h49m13.097838858s) [upload=6997]: Got next segment. Inline: false Aug 7 23:15:59.510819196 (6h49m13.097875417s) [upload=6997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:59.510823413 (6h49m13.097879634s) [upload=6997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:59.498007980 (6h50m13.085064261s) [upload=6997]: Next segment err: context deadline exceeded Aug 7 23:16:59.498313210 (6h50m13.085369461s) [upload=6997]: Done uploading segments... Aug 7 23:16:59.498383411 (6h50m13.085439652s) [upload=6997]: Upload finished. err: context deadline exceeded Duration: 814.322621ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.001978682s FAIL (2025-08-07 23:16:59 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 1:4407 , piece_id=OZ4NVFVFPH3DRK6SZZ5KNKSRH4CJ54PMVAU44IUZCLO5INBGMMCA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487434914 (6h47m1.074491205s) [noise=true , address=104.36.82.9:4026 , piece_id=PA5TEDXTLF2VUXS3ZKJQVN3XA5YOWYTRQ7JHVFJKR6R7FNRKLKQA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487589051 (6h47m1.074645282s) [noise=true , address=104.36.82.44:4877 , piece_id=2DMZUKAPQW3IPUDMQHEHHDY2AKBODJH2HXVLJQLODP65PJRWUU7A, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.487895814 (6h47m1.074952045s) [noise=true , address=45.91.116.4:4959 , piece_id=L4UEVAK4XOTCYG7NJA5ZUSA6M5NVXSLCYWLGS2D4IJ72MBSMHDQQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488059319 (6h47m1.07511555s) [noise=true , address=156.146.43.225:4458 , piece_id=PSBWZ62GHSTTCQNYW5JCUJ3TNGISQTK7UZGBYMM6NJBEO7RNKBOA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488340534 (6h47m1.075396765s) [noise=true , address=104.36.82.8:4996 , piece_id=FL2UEVQWZ3LU7GJLG3DQEUANVU6OOF3TURY6PYRKV4YHV73JNTZA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488536069 (6h47m1.0755923s) [noise=true , address=104.36.82.60:4361 , piece_id=KRZZ7RVKA7OEKNSKRSFDBVFPCO3436LVER5ZYWX2DQGWV66D5WAQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.488721475 (6h47m1.075777716s) [noise=true , address=193.149.164.143:4697 , piece_id=UA3RGUUMVUIJQ2MHCSNNAI6TXQQLYKHYT6D4XUWDJFOCEFRV7OEQ, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.489079183 (6h47m1.076135424s) [noise=true , address=104.36.82.5:4868 , piece_id=VKYGMYQJJATRZV45S4AKZOFDF7ZP66FJNHV7WZY6PATZKNB4F74Q, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.489506110 (6h47m1.07656235s) [noise=true , address=104.36.82.41:4786 , piece_id=CYINWGSZE6EFPKWO5AZKME44HKOPHA22Y2O252GOGJHNGFV2VMNA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.489945269 (6h47m1.077001511s) [noise=true , address=79.127.201.215:4356 , piece_id=6NUG6TPMJMVSOWQTR5WQUE5MDGEBZSNVECQNAUUD5PUN2473HOOA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.490429594 (6h47m1.077485834s) [noise=true , address=104.36.82.9:4040 , piece_id=AFBZHJHUTHJSYXYQQIQ6XTVIFBVO6G5LSA4VARM4JCKJPAXSQS4Q, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.490819892 (6h47m1.077876133s) [noise=true , address=104.36.82.54:4174 , piece_id=4YFSUNEDHTNLMMJOTPMEZBB325UR2VAONVTELFNX6O62UE3AUBUA, seg_pos={0 0}, upload=6952]: Uploading piece... Aug 7 23:13:47.574630371 (6h47m1.161686592s) [upload=6952]: Next returned nil segment Aug 7 23:13:47.574641062 (6h47m1.161697283s) [upload=6952]: Waiting for error group managing segments... Aug 7 23:13:47.587541340 (6h47m1.174597562s) [noise=true , address=104.36.82.41:4786 , piece_id=CYINWGSZE6EFPKWO5AZKME44HKOPHA22Y2O252GOGJHNGFV2VMNA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.592242405 (6h47m1.179298626s) [noise=true , address=104.36.82.39:4748 , piece_id=SJ2OPHKNEGN4YAHZAYB2RHLCNHUFFUQBA7WEQP6BWCBRYWDJIYVQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.599971292 (6h47m1.187027503s) [noise=true , address=104.36.82.9:4040 , piece_id=AFBZHJHUTHJSYXYQQIQ6XTVIFBVO6G5LSA4VARM4JCKJPAXSQS4Q, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.607959633 (6h47m1.195015854s) [noise=true , address=45.91.116.11:4133 , piece_id=ZY3OVRW2WOSIW535LHRPPNPCOE75B26VPK2S27Z6KRKWVDSREWFA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.611555756 (6h47m1.198611967s) [noise=true , address=104.36.82.58:4279 , piece_id=AH5MJW4W346CNJN6LPTKW3FNQV6SUWP5JC47HDH5SRU4D7OWI43Q, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.626811131 (6h47m1.213867352s) [noise=true , address=45.91.116.4:4959 , piece_id=L4UEVAK4XOTCYG7NJA5ZUSA6M5NVXSLCYWLGS2D4IJ72MBSMHDQQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.637742776 (6h47m1.224799007s) [noise=true , address=104.36.82.5:4868 , piece_id=VKYGMYQJJATRZV45S4AKZOFDF7ZP66FJNHV7WZY6PATZKNB4F74Q, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.639233228 (6h47m1.226289449s) [noise=true , address=104.36.82.60:4361 , piece_id=KRZZ7RVKA7OEKNSKRSFDBVFPCO3436LVER5ZYWX2DQGWV66D5WAQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.639680683 (6h47m1.226736904s) [noise=true , address=104.36.82.44:4877 , piece_id=2DMZUKAPQW3IPUDMQHEHHDY2AKBODJH2HXVLJQLODP65PJRWUU7A, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.646508387 (6h47m1.233564599s) [noise=true , address=104.36.82.54:4174 , piece_id=4YFSUNEDHTNLMMJOTPMEZBB325UR2VAONVTELFNX6O62UE3AUBUA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.669751322 (6h47m1.256807543s) [noise=true , address=104.36.82.8:4996 , piece_id=FL2UEVQWZ3LU7GJLG3DQEUANVU6OOF3TURY6PYRKV4YHV73JNTZA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.722599178 (6h47m1.309655429s) [noise=true , address=79.127.201.215:4359 , piece_id=ROGWPZSHIMRJHEF7EHHDFH2E4VZQBE5ESC6EJV623NMGE7PUGW2A, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.724255018 (6h47m1.31131124s) [noise=true , address=156.146.43.225:4458 , piece_id=PSBWZ62GHSTTCQNYW5JCUJ3TNGISQTK7UZGBYMM6NJBEO7RNKBOA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.740407329 (6h47m1.32746358s) [noise=true , address=79.127.201.215:4356 , piece_id=6NUG6TPMJMVSOWQTR5WQUE5MDGEBZSNVECQNAUUD5PUN2473HOOA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: <nil> Aug 7 23:13:47.740458694 (6h47m1.327514935s) [seg_pos={0 0}, upload=6952]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:47.740645193 (6h47m1.327701414s) [noise=true , address=104.36.82.9:4026 , piece_id=PA5TEDXTLF2VUXS3ZKJQVN3XA5YOWYTRQ7JHVFJKR6R7FNRKLKQA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1M7HJBLoxHkRHXbPuDiqwxdb4EAq5wktfJY5FygdoDdgy59rRq): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:47.740952506 (6h47m1.328008727s) [noise=true , address=64.49.13.169:4713 , piece_id=UOCWLHFGT2MI3K4IIDO3YIFJIXZGMCRWF55QSBRIKDZD5DSBQDVA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: read tcp 5.161.89.252:25140->64.49.13.169:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:47.741048936 (6h47m1.328105147s) [noise=true , address=104.36.82.61:4407 , piece_id=OZ4NVFVFPH3DRK6SZZ5KNKSRH4CJ54PMVAU44IUZCLO5INBGMMCA, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122k7b6wa2JhSXMVAhF4qPKA48cWkh3DZRyWuwPEt7Vp3ciofPF): protocol: read tcp 5.161.89.252:36630->104.36.82.61:4407: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:47.741097086 (6h47m1.328153327s) [noise=true , address=193.149.164.143:4697 , piece_id=UA3RGUUMVUIJQ2MHCSNNAI6TXQQLYKHYT6D4XUWDJFOCEFRV7OEQ, seg_pos={0 0}, upload=6952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: read tcp 5.161.89.252:22104->193.149.164.143:4697: use of closed network connection; conn closed: op Aug 7 23:13:47.741147290 (6h47m1.32820353s) [seg_pos={0 0}, upload=6952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:47.741286780 (6h47m1.328343021s) [upload=6952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:47.741297580 (6h47m1.328353832s) [upload=6952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:59.351303634 (6h48m12.938359875s) [upload=6952]: Done uploading segments... Aug 7 23:14:59.351331766 (6h48m12.938388007s) [upload=6952]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:59.433768531 (6h48m13.020824773s) [upload=6982]: Starting upload Aug 7 23:14:59.433862216 (6h48m13.020918457s) [upload=6982]: Done starting upload Aug 7 23:14:59.434026252 (6h48m13.021082504s) [upload=6982]: Uploading segments... Aug 7 23:14:59.434316203 (6h48m13.021372445s) [upload=6982]: Got next segment. Inline: false Aug 7 23:14:59.434383048 (6h48m13.02143929s) [upload=6982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.434389380 (6h48m13.021445622s) [upload=6982]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:59.416073113 (6h49m13.003129334s) [upload=6982]: Next segment err: context deadline exceeded Aug 7 23:15:59.416143385 (6h49m13.003199606s) [upload=6982]: Done uploading segments... Aug 7 23:15:59.416158513 (6h49m13.003214734s) [upload=6982]: Upload finished. err: context deadline exceeded Aug 7 23:15:59.510497796 (6h49m13.097554017s) [upload=6997]: Starting upload Aug 7 23:15:59.510549312 (6h49m13.097605533s) [upload=6997]: Done starting upload Aug 7 23:15:59.510663394 (6h49m13.097719606s) [upload=6997]: Uploading segments... Aug 7 23:15:59.510782647 (6h49m13.097838858s) [upload=6997]: Got next segment. Inline: false Aug 7 23:15:59.510819196 (6h49m13.097875417s) [upload=6997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:59.510823413 (6h49m13.097879634s) [upload=6997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:59.498007980 (6h50m13.085064261s) [upload=6997]: Next segment err: context deadline exceeded Aug 7 23:16:59.498313210 (6h50m13.085369461s) [upload=6997]: Done uploading segments... Aug 7 23:16:59.498383411 (6h50m13.085439652s) [upload=6997]: Upload finished. err: context deadline exceeded Duration: 1m0.001997236s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X.......................................................XXXX] duration: 1m0.001660159s FAIL (2025-08-07 23:16:57 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001681799s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)