Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 163.584777ms FAIL (2025-08-07 23:57:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 163.605787ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 10 out of 60 is failed (threshold=3) [.XXXXXX.X...............................................XX.X] duration: 1m0.000752295s FAIL (2025-08-07 23:56:57 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:39.988158721 (7h20m53.575214932s) [upload=7536]: Done uploading segments... Aug 7 23:47:39.988176785 (7h20m53.575232995s) [upload=7536]: Upload finished. err: <nil> Aug 7 23:48:17.585904864 (7h21m31.172961076s) [upload=7548]: Starting upload Aug 7 23:48:17.585990855 (7h21m31.173047066s) [upload=7548]: Done starting upload Aug 7 23:48:17.586065835 (7h21m31.173122035s) [upload=7548]: Uploading segments... Aug 7 23:48:17.586102964 (7h21m31.173159205s) [upload=7548]: Got next segment. Inline: true Aug 7 23:48:17.586107262 (7h21m31.173163473s) [upload=7548]: Waiting for error group managing segments... Aug 7 23:48:17.586144451 (7h21m31.173200662s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:17.586148969 (7h21m31.17320518s) [upload=7548]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:17.586151854 (7h21m31.173208066s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:17.642355873 (7h21m31.229412094s) [upload=7548]: Done uploading segments... Aug 7 23:48:17.642371582 (7h21m31.229427794s) [upload=7548]: Upload finished. err: <nil> Aug 7 23:49:19.929340015 (7h22m33.516396255s) [upload=7565]: Starting upload Aug 7 23:49:19.929497669 (7h22m33.51655391s) [upload=7565]: Done starting upload Aug 7 23:49:19.929593468 (7h22m33.516649709s) [upload=7565]: Uploading segments... Aug 7 23:49:19.929640947 (7h22m33.516697188s) [upload=7565]: Got next segment. Inline: true Aug 7 23:49:19.929647679 (7h22m33.516703931s) [upload=7565]: Waiting for error group managing segments... Aug 7 23:49:19.929695078 (7h22m33.516751329s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:19.929701059 (7h22m33.5167573s) [upload=7565]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:19.929704956 (7h22m33.516761197s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:19.992636681 (7h22m33.579692922s) [upload=7565]: Done uploading segments... Aug 7 23:49:19.992654385 (7h22m33.579710625s) [upload=7565]: Upload finished. err: <nil> Aug 7 23:50:07.176468149 (7h23m20.76352438s) [upload=7576]: Starting upload Aug 7 23:50:07.176537798 (7h23m20.763594019s) [upload=7576]: Done starting upload Aug 7 23:50:07.176592070 (7h23m20.763648311s) [upload=7576]: Uploading segments... Aug 7 23:50:07.176626473 (7h23m20.763682705s) [upload=7576]: Got next segment. Inline: true Aug 7 23:50:07.176631002 (7h23m20.763687213s) [upload=7576]: Waiting for error group managing segments... Aug 7 23:50:07.176664084 (7h23m20.763720295s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:07.176668132 (7h23m20.763724343s) [upload=7576]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:07.176670586 (7h23m20.763726807s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:07.240851653 (7h23m20.827907864s) [upload=7576]: Done uploading segments... Aug 7 23:50:07.240866771 (7h23m20.827922992s) [upload=7576]: Upload finished. err: <nil> Aug 7 23:50:45.893754262 (7h23m59.480810504s) [upload=7591]: Starting upload Aug 7 23:50:45.893825847 (7h23m59.480882088s) [upload=7591]: Done starting upload Aug 7 23:50:45.893912749 (7h23m59.48096899s) [upload=7591]: Uploading segments... Aug 7 23:50:45.893955949 (7h23m59.4810122s) [upload=7591]: Got next segment. Inline: true Aug 7 23:50:45.893961970 (7h23m59.481018211s) [upload=7591]: Waiting for error group managing segments... Aug 7 23:50:45.894005482 (7h23m59.481061723s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:45.894011062 (7h23m59.481067304s) [upload=7591]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:45.894014729 (7h23m59.48107097s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:05.079621048 (7h24m18.666677259s) [upload=7591]: Done uploading segments... Aug 7 23:51:05.079636297 (7h24m18.666692508s) [upload=7591]: Upload finished. err: <nil> Aug 7 23:51:21.725622238 (7h24m35.312678469s) [upload=7603]: Starting upload Aug 7 23:51:21.725696627 (7h24m35.312752838s) [upload=7603]: Done starting upload Aug 7 23:51:21.725758231 (7h24m35.312814443s) [upload=7603]: Uploading segments... Aug 7 23:51:21.725796774 (7h24m35.312852985s) [upload=7603]: Got next segment. Inline: true Aug 7 23:51:21.725802845 (7h24m35.312859056s) [upload=7603]: Waiting for error group managing segments... Aug 7 23:51:21.725848871 (7h24m35.312905082s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:21.725855313 (7h24m35.312911523s) [upload=7603]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:21.725859100 (7h24m35.312915351s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:21.784505948 (7h24m35.371562158s) [upload=7603]: Done uploading segments... Aug 7 23:51:21.784521767 (7h24m35.371577978s) [upload=7603]: Upload finished. err: <nil> Aug 7 23:52:10.285208176 (7h25m23.872264397s) [upload=7615]: Starting upload Aug 7 23:52:10.285263228 (7h25m23.872319449s) [upload=7615]: Done starting upload Aug 7 23:52:10.285310727 (7h25m23.872366937s) [upload=7615]: Uploading segments... Aug 7 23:52:10.285339711 (7h25m23.872395932s) [upload=7615]: Got next segment. Inline: true Aug 7 23:52:10.285343438 (7h25m23.872399649s) [upload=7615]: Waiting for error group managing segments... Aug 7 23:52:10.285391158 (7h25m23.872447379s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:10.285395065 (7h25m23.872451276s) [upload=7615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:10.285397770 (7h25m23.872453991s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:10.400888570 (7h25m23.987944781s) [upload=7615]: Done uploading segments... Aug 7 23:52:10.400904530 (7h25m23.987960741s) [upload=7615]: Upload finished. err: <nil> Aug 7 23:53:11.795016020 (7h26m25.382072262s) [upload=7635]: Starting upload Aug 7 23:53:11.795108102 (7h26m25.382164354s) [upload=7635]: Done starting upload Aug 7 23:53:11.795175588 (7h26m25.382231829s) [upload=7635]: Uploading segments... Aug 7 23:53:11.795254645 (7h26m25.382310886s) [upload=7635]: Got next segment. Inline: true Aug 7 23:53:11.795262130 (7h26m25.38231837s) [upload=7635]: Waiting for error group managing segments... Aug 7 23:53:11.795319296 (7h26m25.382375538s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:11.795326170 (7h26m25.38238242s) [upload=7635]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:11.795329857 (7h26m25.382386097s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:11.782974743 (7h27m25.370030954s) [upload=7635]: Done uploading segments... Aug 7 23:54:11.782991474 (7h27m25.370047695s) [upload=7635]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:11.855031501 (7h27m25.442087722s) [upload=7650]: Starting upload Aug 7 23:54:11.855087627 (7h27m25.442143838s) [upload=7650]: Done starting upload Aug 7 23:54:11.855131608 (7h27m25.442187829s) [upload=7650]: Uploading segments... Aug 7 23:54:11.855163227 (7h27m25.442219448s) [upload=7650]: Got next segment. Inline: true Aug 7 23:54:11.855167005 (7h27m25.442223226s) [upload=7650]: Waiting for error group managing segments... Aug 7 23:54:11.855198905 (7h27m25.442255126s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:11.855202661 (7h27m25.442258882s) [upload=7650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:11.855205036 (7h27m25.442261247s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:11.844108363 (7h28m25.431164634s) [upload=7650]: Done uploading segments... Aug 7 23:55:11.844129542 (7h28m25.431185763s) [upload=7650]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:11.924363315 (7h28m25.511419527s) [upload=7668]: Starting upload Aug 7 23:55:11.924429198 (7h28m25.511485409s) [upload=7668]: Done starting upload Aug 7 23:55:11.924484862 (7h28m25.511541083s) [upload=7668]: Uploading segments... Aug 7 23:55:11.924519887 (7h28m25.511576098s) [upload=7668]: Got next segment. Inline: true Aug 7 23:55:11.924523564 (7h28m25.511579785s) [upload=7668]: Waiting for error group managing segments... Aug 7 23:55:11.924554131 (7h28m25.511610342s) [upload=7668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:11.924558279 (7h28m25.51161449s) [upload=7668]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:11.924560643 (7h28m25.511616855s) [upload=7668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:12.020746910 (7h28m25.607803121s) [upload=7668]: Done uploading segments... Aug 7 23:55:12.020765404 (7h28m25.607821625s) [upload=7668]: Upload finished. err: <nil> Aug 7 23:55:57.071847012 (7h29m10.658903254s) [upload=7678]: Starting upload Aug 7 23:55:57.071985611 (7h29m10.659041851s) [upload=7678]: Done starting upload Aug 7 23:55:57.072082963 (7h29m10.659139204s) [upload=7678]: Uploading segments... Aug 7 23:55:57.072231269 (7h29m10.65928751s) [upload=7678]: Got next segment. Inline: true Aug 7 23:55:57.072240506 (7h29m10.659296748s) [upload=7678]: Waiting for error group managing segments... Aug 7 23:55:57.072359458 (7h29m10.6594157s) [upload=7678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:57.072367143 (7h29m10.659423384s) [upload=7678]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:57.072371320 (7h29m10.659427562s) [upload=7678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:57.049680168 (7h30m10.636736379s) [upload=7678]: Done uploading segments... Aug 7 23:56:57.049696499 (7h30m10.636752709s) [upload=7678]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1m0.000765641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 667.511227ms FAIL (2025-08-07 23:56:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== 1345s) [upload=7262]: Got next segment. Inline: false Aug 7 23:31:37.583906637 (7h4m51.170962878s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:31:37.583953414 (7h4m51.171009656s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:31:37.613963041 (7h4m51.201019262s) [seg_pos={0 0}, upload=7262]: Begin upload segment... Aug 7 23:31:37.614008966 (7h4m51.201065176s) [seg_pos={0 0}, upload=7262]: Done begin upload segment. Aug 7 23:31:37.614029103 (7h4m51.201085324s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614108311 (7h4m51.201164542s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614344612 (7h4m51.201400833s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614634754 (7h4m51.201690975s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614919926 (7h4m51.201976137s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614943150 (7h4m51.201999391s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615146719 (7h4m51.20220294s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615454163 (7h4m51.202510384s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615514677 (7h4m51.202570918s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615688160 (7h4m51.202744381s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615932236 (7h4m51.202988457s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616184166 (7h4m51.203240378s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616521716 (7h4m51.203577937s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616643614 (7h4m51.203699855s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621231117 (7h4m51.208287338s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621561163 (7h4m51.208617374s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621861323 (7h4m51.208917534s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.622156223 (7h4m51.209212444s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.726040884 (7h4m51.313097094s) [upload=7262]: Next returned nil segment Aug 7 23:31:37.726066772 (7h4m51.313122983s) [upload=7262]: Waiting for error group managing segments... Aug 7 23:31:37.736232517 (7h4m51.323288738s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.737656014 (7h4m51.324712245s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.739754932 (7h4m51.326811153s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.740628113 (7h4m51.327684344s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741712657 (7h4m51.328768867s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741742653 (7h4m51.328798883s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.748242274 (7h4m51.335298505s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.750434577 (7h4m51.337490798s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.758962364 (7h4m51.346018586s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.769693706 (7h4m51.356749917s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.800540734 (7h4m51.387596975s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.924012655 (7h4m51.511068866s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.059695219 (7h4m51.64675144s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102515917 (7h4m51.689572138s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102541606 (7h4m51.689597827s) [seg_pos={0 0}, upload=7262]: Segment reached optimal threshold of 14 pieces. Aug 7 23:31:38.102658574 (7h4m51.689714795s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF Aug 7 23:31:38.102748111 (7h4m51.689804332s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE): protocol: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection Aug 7 23:31:38.102832158 (7h4m51.689888399s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GcyWwKkfavsCsmtq55U72n6XeZKqQiGGYHY9D6UwST8sC14Ac): protocol: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection Aug 7 23:31:38.102967200 (7h4m51.690023421s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12syasUPsyYf4UstQ6cFAr6qamze3vCyAm9dnobk5e83w4bTz69): protocol: context canceled; context canceled Aug 7 23:31:38.102995132 (7h4m51.690051374s) [seg_pos={0 0}, upload=7262]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:31:38.103097553 (7h4m51.690153775s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:31:38.103102412 (7h4m51.690158633s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:31:38.203633212 (7h4m51.790689433s) [upload=7262]: Done uploading segments... Aug 7 23:31:38.203649102 (7h4m51.790705323s) [upload=7262]: Upload finished. err: <nil> Duration: 667.540172ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 14 out of 60 is failed (threshold=3) [XXXX.X..X..X......................................XXX.X.XXX.] duration: 204.572178ms FAIL (2025-08-07 23:56:54 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 204.588859ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 14 out of 60 is failed (threshold=3) [.XXXX.X..X..X......................................XXX.X.XXX] duration: 30.000568053s FAIL (2025-08-07 23:56:53 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000590214s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [..X.XXXXX..X................................................] duration: 185.921716ms FAIL (2025-08-07 23:56:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 185.94485ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 15 out of 60 is failed (threshold=3) [XXXXX.XX............................................XXXXXXXX] duration: 30.001830394s FAIL (2025-08-07 23:56:45 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Aug 7 23:48:58.840462661 (7h22m12.427518891s) [upload=7557]: Upload finished. err: <nil> Aug 7 23:50:01.971851413 (7h23m15.558907624s) [upload=7574]: Starting upload Aug 7 23:50:01.971890696 (7h23m15.558946907s) [upload=7574]: Done starting upload Aug 7 23:50:01.971977749 (7h23m15.559033961s) [upload=7574]: Uploading segments... Aug 7 23:50:01.972022943 (7h23m15.559079154s) [upload=7574]: Got next segment. Inline: true Aug 7 23:50:01.972026850 (7h23m15.559083062s) [upload=7574]: Waiting for error group managing segments... Aug 7 23:50:01.972060975 (7h23m15.559117185s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:01.972064511 (7h23m15.559120712s) [upload=7574]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:01.972066685 (7h23m15.559122896s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:02.063709030 (7h23m15.650765241s) [upload=7574]: Done uploading segments... Aug 7 23:50:02.063723547 (7h23m15.650779768s) [upload=7574]: Upload finished. err: <nil> Aug 7 23:50:41.160166556 (7h23m54.747222777s) [upload=7588]: Starting upload Aug 7 23:50:41.160220555 (7h23m54.747276777s) [upload=7588]: Done starting upload Aug 7 23:50:41.160264659 (7h23m54.74732088s) [upload=7588]: Uploading segments... Aug 7 23:50:41.160295956 (7h23m54.747352177s) [upload=7588]: Got next segment. Inline: true Aug 7 23:50:41.160299753 (7h23m54.747355964s) [upload=7588]: Waiting for error group managing segments... Aug 7 23:50:41.160331633 (7h23m54.747387844s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:41.160335861 (7h23m54.747392072s) [upload=7588]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:41.160338396 (7h23m54.747394617s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:41.222859735 (7h23m54.809915976s) [upload=7588]: Done uploading segments... Aug 7 23:50:41.222879913 (7h23m54.809936154s) [upload=7588]: Upload finished. err: <nil> Aug 7 23:51:17.131690986 (7h24m30.718747197s) [upload=7599]: Starting upload Aug 7 23:51:17.131738926 (7h24m30.718795147s) [upload=7599]: Done starting upload Aug 7 23:51:17.131792947 (7h24m30.718849178s) [upload=7599]: Uploading segments... Aug 7 23:51:17.131835896 (7h24m30.718892117s) [upload=7599]: Got next segment. Inline: true Aug 7 23:51:17.131842027 (7h24m30.718898239s) [upload=7599]: Waiting for error group managing segments... Aug 7 23:51:17.131883385 (7h24m30.718939627s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:17.131889146 (7h24m30.718945388s) [upload=7599]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:17.132006205 (7h24m30.719062435s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:38.787433326 (7h24m52.374489567s) [upload=7599]: Done uploading segments... Aug 7 23:51:38.787452212 (7h24m52.374508453s) [upload=7599]: Upload finished. err: <nil> Aug 7 23:52:00.256204647 (7h25m13.843260858s) [upload=7610]: Starting upload Aug 7 23:52:00.256263547 (7h25m13.843319758s) [upload=7610]: Done starting upload Aug 7 23:52:00.256316195 (7h25m13.843372406s) [upload=7610]: Uploading segments... Aug 7 23:52:00.256345671 (7h25m13.843401881s) [upload=7610]: Got next segment. Inline: true Aug 7 23:52:00.256349758 (7h25m13.843405969s) [upload=7610]: Waiting for error group managing segments... Aug 7 23:52:00.256399992 (7h25m13.843456203s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:00.256404780 (7h25m13.843460991s) [upload=7610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:00.256407806 (7h25m13.843464017s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:30.237600164 (7h25m43.824656416s) [upload=7610]: Done uploading segments... Aug 7 23:52:30.237630862 (7h25m43.824687113s) [upload=7610]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:53.146251246 (7h26m6.733307456s) [upload=7626]: Starting upload Aug 7 23:52:53.146307339 (7h26m6.733363561s) [upload=7626]: Done starting upload Aug 7 23:52:53.146350962 (7h26m6.733407173s) [upload=7626]: Uploading segments... Aug 7 23:52:53.146411344 (7h26m6.733467565s) [upload=7626]: Got next segment. Inline: true Aug 7 23:52:53.146415271 (7h26m6.733471493s) [upload=7626]: Waiting for error group managing segments... Aug 7 23:52:53.146449966 (7h26m6.733506177s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:53.146454174 (7h26m6.733510395s) [upload=7626]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:53.146456709 (7h26m6.73351293s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.132536521 (7h26m36.719592742s) [upload=7626]: Done uploading segments... Aug 7 23:53:23.132560907 (7h26m36.719617128s) [upload=7626]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:53:28.217132518 (7h26m41.804188759s) [upload=7638]: Starting upload Aug 7 23:53:28.217216185 (7h26m41.804272426s) [upload=7638]: Done starting upload Aug 7 23:53:28.217288009 (7h26m41.80434426s) [upload=7638]: Uploading segments... Aug 7 23:53:28.217336129 (7h26m41.80439237s) [upload=7638]: Got next segment. Inline: true Aug 7 23:53:28.217341890 (7h26m41.804398131s) [upload=7638]: Waiting for error group managing segments... Aug 7 23:53:28.217421438 (7h26m41.804477679s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:28.217427650 (7h26m41.804483901s) [upload=7638]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:28.217431617 (7h26m41.804487859s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:58.204830989 (7h27m11.79188721s) [upload=7638]: Done uploading segments... Aug 7 23:53:58.204849294 (7h27m11.791905505s) [upload=7638]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:01.364416813 (7h27m14.951473054s) [upload=7648]: Starting upload Aug 7 23:54:01.364544481 (7h27m14.951600732s) [upload=7648]: Done starting upload Aug 7 23:54:01.364639518 (7h27m14.95169577s) [upload=7648]: Uploading segments... Aug 7 23:54:01.364703858 (7h27m14.951760109s) [upload=7648]: Got next segment. Inline: true Aug 7 23:54:01.364713517 (7h27m14.951769757s) [upload=7648]: Waiting for error group managing segments... Aug 7 23:54:01.364781374 (7h27m14.951837625s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:01.364791231 (7h27m14.951847472s) [upload=7648]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:01.364797502 (7h27m14.951853754s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:31.351186626 (7h27m44.938242847s) [upload=7648]: Done uploading segments... Aug 7 23:54:31.351241118 (7h27m44.938297339s) [upload=7648]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:31.416140839 (7h27m45.003197049s) [upload=7656]: Starting upload Aug 7 23:54:31.416189309 (7h27m45.00324553s) [upload=7656]: Done starting upload Aug 7 23:54:31.416235946 (7h27m45.003292157s) [upload=7656]: Uploading segments... Aug 7 23:54:31.416277384 (7h27m45.003333595s) [upload=7656]: Got next segment. Inline: true Aug 7 23:54:31.416297601 (7h27m45.003353813s) [upload=7656]: Waiting for error group managing segments... Aug 7 23:54:31.416336454 (7h27m45.003392665s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:31.416340652 (7h27m45.003396873s) [upload=7656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:31.416342965 (7h27m45.003399177s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:01.405081824 (7h28m14.992138035s) [upload=7656]: Done uploading segments... Aug 7 23:55:01.405101451 (7h28m14.992157672s) [upload=7656]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:45.764200449 (7h28m59.35125669s) [upload=7673]: Starting upload Aug 7 23:55:45.764300708 (7h28m59.351356948s) [upload=7673]: Done starting upload Aug 7 23:55:45.764398370 (7h28m59.351454611s) [upload=7673]: Uploading segments... Aug 7 23:55:45.764515538 (7h28m59.35157178s) [upload=7673]: Got next segment. Inline: true Aug 7 23:55:45.764521980 (7h28m59.351578222s) [upload=7673]: Waiting for error group managing segments... Aug 7 23:55:45.764570060 (7h28m59.3516263s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:45.764575871 (7h28m59.351632111s) [upload=7673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:45.764579898 (7h28m59.351636149s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:15.747212127 (7h29m29.334268369s) [upload=7673]: Done uploading segments... Aug 7 23:56:15.747307315 (7h29m29.334363566s) [upload=7673]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:15.823083783 (7h29m29.410140035s) [upload=7682]: Starting upload Aug 7 23:56:15.823169003 (7h29m29.410225243s) [upload=7682]: Done starting upload Aug 7 23:56:15.823245125 (7h29m29.410301367s) [upload=7682]: Uploading segments... Aug 7 23:56:15.823289758 (7h29m29.410346009s) [upload=7682]: Got next segment. Inline: true Aug 7 23:56:15.823295359 (7h29m29.410351599s) [upload=7682]: Waiting for error group managing segments... Aug 7 23:56:15.823343018 (7h29m29.410399269s) [upload=7682]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:15.823348678 (7h29m29.41040492s) [upload=7682]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:15.823352376 (7h29m29.410408616s) [upload=7682]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:45.810149218 (7h29m59.397205439s) [upload=7682]: Done uploading segments... Aug 7 23:56:45.810167081 (7h29m59.397223322s) [upload=7682]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001844761s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX.X....X.X..........................................X.....] duration: 562.269088ms FAIL (2025-08-07 23:56:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 562.28632ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.XXXXXX.X.X...........................................X.X..X] duration: 30.000364608s FAIL (2025-08-07 23:56:21 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000385067s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 13 out of 60 is failed (threshold=3) [..XXXX.X..X..X......................................XXX.X.XX] duration: 30.000410383s FAIL (2025-08-07 23:56:16 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000432884s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 14 out of 60 is failed (threshold=3) [.XXXXX.XX............................................XXXXXXX] duration: 30.00150159s FAIL (2025-08-07 23:56:15 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad=7545]: Done uploading segments... Aug 7 23:48:09.951286926 (7h21m23.538343148s) [upload=7545]: Upload finished. err: <nil> Aug 7 23:48:58.772502770 (7h22m12.359559011s) [upload=7557]: Starting upload Aug 7 23:48:58.772574805 (7h22m12.359631046s) [upload=7557]: Done starting upload Aug 7 23:48:58.772645527 (7h22m12.359701767s) [upload=7557]: Uploading segments... Aug 7 23:48:58.772695300 (7h22m12.359751541s) [upload=7557]: Got next segment. Inline: true Aug 7 23:48:58.772701031 (7h22m12.359757271s) [upload=7557]: Waiting for error group managing segments... Aug 7 23:48:58.772770249 (7h22m12.359826501s) [upload=7557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:58.772780228 (7h22m12.359836469s) [upload=7557]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:58.772787010 (7h22m12.359843252s) [upload=7557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:58.840436652 (7h22m12.427492893s) [upload=7557]: Done uploading segments... Aug 7 23:48:58.840462661 (7h22m12.427518891s) [upload=7557]: Upload finished. err: <nil> Aug 7 23:50:01.971851413 (7h23m15.558907624s) [upload=7574]: Starting upload Aug 7 23:50:01.971890696 (7h23m15.558946907s) [upload=7574]: Done starting upload Aug 7 23:50:01.971977749 (7h23m15.559033961s) [upload=7574]: Uploading segments... Aug 7 23:50:01.972022943 (7h23m15.559079154s) [upload=7574]: Got next segment. Inline: true Aug 7 23:50:01.972026850 (7h23m15.559083062s) [upload=7574]: Waiting for error group managing segments... Aug 7 23:50:01.972060975 (7h23m15.559117185s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:01.972064511 (7h23m15.559120712s) [upload=7574]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:01.972066685 (7h23m15.559122896s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:02.063709030 (7h23m15.650765241s) [upload=7574]: Done uploading segments... Aug 7 23:50:02.063723547 (7h23m15.650779768s) [upload=7574]: Upload finished. err: <nil> Aug 7 23:50:41.160166556 (7h23m54.747222777s) [upload=7588]: Starting upload Aug 7 23:50:41.160220555 (7h23m54.747276777s) [upload=7588]: Done starting upload Aug 7 23:50:41.160264659 (7h23m54.74732088s) [upload=7588]: Uploading segments... Aug 7 23:50:41.160295956 (7h23m54.747352177s) [upload=7588]: Got next segment. Inline: true Aug 7 23:50:41.160299753 (7h23m54.747355964s) [upload=7588]: Waiting for error group managing segments... Aug 7 23:50:41.160331633 (7h23m54.747387844s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:41.160335861 (7h23m54.747392072s) [upload=7588]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:41.160338396 (7h23m54.747394617s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:41.222859735 (7h23m54.809915976s) [upload=7588]: Done uploading segments... Aug 7 23:50:41.222879913 (7h23m54.809936154s) [upload=7588]: Upload finished. err: <nil> Aug 7 23:51:17.131690986 (7h24m30.718747197s) [upload=7599]: Starting upload Aug 7 23:51:17.131738926 (7h24m30.718795147s) [upload=7599]: Done starting upload Aug 7 23:51:17.131792947 (7h24m30.718849178s) [upload=7599]: Uploading segments... Aug 7 23:51:17.131835896 (7h24m30.718892117s) [upload=7599]: Got next segment. Inline: true Aug 7 23:51:17.131842027 (7h24m30.718898239s) [upload=7599]: Waiting for error group managing segments... Aug 7 23:51:17.131883385 (7h24m30.718939627s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:17.131889146 (7h24m30.718945388s) [upload=7599]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:17.132006205 (7h24m30.719062435s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:38.787433326 (7h24m52.374489567s) [upload=7599]: Done uploading segments... Aug 7 23:51:38.787452212 (7h24m52.374508453s) [upload=7599]: Upload finished. err: <nil> Aug 7 23:52:00.256204647 (7h25m13.843260858s) [upload=7610]: Starting upload Aug 7 23:52:00.256263547 (7h25m13.843319758s) [upload=7610]: Done starting upload Aug 7 23:52:00.256316195 (7h25m13.843372406s) [upload=7610]: Uploading segments... Aug 7 23:52:00.256345671 (7h25m13.843401881s) [upload=7610]: Got next segment. Inline: true Aug 7 23:52:00.256349758 (7h25m13.843405969s) [upload=7610]: Waiting for error group managing segments... Aug 7 23:52:00.256399992 (7h25m13.843456203s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:00.256404780 (7h25m13.843460991s) [upload=7610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:00.256407806 (7h25m13.843464017s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:30.237600164 (7h25m43.824656416s) [upload=7610]: Done uploading segments... Aug 7 23:52:30.237630862 (7h25m43.824687113s) [upload=7610]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:53.146251246 (7h26m6.733307456s) [upload=7626]: Starting upload Aug 7 23:52:53.146307339 (7h26m6.733363561s) [upload=7626]: Done starting upload Aug 7 23:52:53.146350962 (7h26m6.733407173s) [upload=7626]: Uploading segments... Aug 7 23:52:53.146411344 (7h26m6.733467565s) [upload=7626]: Got next segment. Inline: true Aug 7 23:52:53.146415271 (7h26m6.733471493s) [upload=7626]: Waiting for error group managing segments... Aug 7 23:52:53.146449966 (7h26m6.733506177s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:53.146454174 (7h26m6.733510395s) [upload=7626]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:53.146456709 (7h26m6.73351293s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.132536521 (7h26m36.719592742s) [upload=7626]: Done uploading segments... Aug 7 23:53:23.132560907 (7h26m36.719617128s) [upload=7626]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:53:28.217132518 (7h26m41.804188759s) [upload=7638]: Starting upload Aug 7 23:53:28.217216185 (7h26m41.804272426s) [upload=7638]: Done starting upload Aug 7 23:53:28.217288009 (7h26m41.80434426s) [upload=7638]: Uploading segments... Aug 7 23:53:28.217336129 (7h26m41.80439237s) [upload=7638]: Got next segment. Inline: true Aug 7 23:53:28.217341890 (7h26m41.804398131s) [upload=7638]: Waiting for error group managing segments... Aug 7 23:53:28.217421438 (7h26m41.804477679s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:28.217427650 (7h26m41.804483901s) [upload=7638]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:28.217431617 (7h26m41.804487859s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:58.204830989 (7h27m11.79188721s) [upload=7638]: Done uploading segments... Aug 7 23:53:58.204849294 (7h27m11.791905505s) [upload=7638]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:01.364416813 (7h27m14.951473054s) [upload=7648]: Starting upload Aug 7 23:54:01.364544481 (7h27m14.951600732s) [upload=7648]: Done starting upload Aug 7 23:54:01.364639518 (7h27m14.95169577s) [upload=7648]: Uploading segments... Aug 7 23:54:01.364703858 (7h27m14.951760109s) [upload=7648]: Got next segment. Inline: true Aug 7 23:54:01.364713517 (7h27m14.951769757s) [upload=7648]: Waiting for error group managing segments... Aug 7 23:54:01.364781374 (7h27m14.951837625s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:01.364791231 (7h27m14.951847472s) [upload=7648]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:01.364797502 (7h27m14.951853754s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:31.351186626 (7h27m44.938242847s) [upload=7648]: Done uploading segments... Aug 7 23:54:31.351241118 (7h27m44.938297339s) [upload=7648]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:31.416140839 (7h27m45.003197049s) [upload=7656]: Starting upload Aug 7 23:54:31.416189309 (7h27m45.00324553s) [upload=7656]: Done starting upload Aug 7 23:54:31.416235946 (7h27m45.003292157s) [upload=7656]: Uploading segments... Aug 7 23:54:31.416277384 (7h27m45.003333595s) [upload=7656]: Got next segment. Inline: true Aug 7 23:54:31.416297601 (7h27m45.003353813s) [upload=7656]: Waiting for error group managing segments... Aug 7 23:54:31.416336454 (7h27m45.003392665s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:31.416340652 (7h27m45.003396873s) [upload=7656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:31.416342965 (7h27m45.003399177s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:01.405081824 (7h28m14.992138035s) [upload=7656]: Done uploading segments... Aug 7 23:55:01.405101451 (7h28m14.992157672s) [upload=7656]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:45.764200449 (7h28m59.35125669s) [upload=7673]: Starting upload Aug 7 23:55:45.764300708 (7h28m59.351356948s) [upload=7673]: Done starting upload Aug 7 23:55:45.764398370 (7h28m59.351454611s) [upload=7673]: Uploading segments... Aug 7 23:55:45.764515538 (7h28m59.35157178s) [upload=7673]: Got next segment. Inline: true Aug 7 23:55:45.764521980 (7h28m59.351578222s) [upload=7673]: Waiting for error group managing segments... Aug 7 23:55:45.764570060 (7h28m59.3516263s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:45.764575871 (7h28m59.351632111s) [upload=7673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:45.764579898 (7h28m59.351636149s) [upload=7673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:15.747212127 (7h29m29.334268369s) [upload=7673]: Done uploading segments... Aug 7 23:56:15.747307315 (7h29m29.334363566s) [upload=7673]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001527678s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [....XXXXX..X............................................XXXX] duration: 1m0.000632978s FAIL (2025-08-07 23:56:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000660942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 159.510726ms FAIL (2025-08-07 23:56:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 159.543086ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [...X.XXXXX..X...............................................] duration: 172.546725ms FAIL (2025-08-07 23:55:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 172.568827ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.XX.X....X.X..........................................X....] duration: 416.867638ms FAIL (2025-08-07 23:55:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 416.886053ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.090907317s FAIL (2025-08-07 23:55:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== 1345s) [upload=7262]: Got next segment. Inline: false Aug 7 23:31:37.583906637 (7h4m51.170962878s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:31:37.583953414 (7h4m51.171009656s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:31:37.613963041 (7h4m51.201019262s) [seg_pos={0 0}, upload=7262]: Begin upload segment... Aug 7 23:31:37.614008966 (7h4m51.201065176s) [seg_pos={0 0}, upload=7262]: Done begin upload segment. Aug 7 23:31:37.614029103 (7h4m51.201085324s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614108311 (7h4m51.201164542s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614344612 (7h4m51.201400833s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614634754 (7h4m51.201690975s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614919926 (7h4m51.201976137s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614943150 (7h4m51.201999391s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615146719 (7h4m51.20220294s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615454163 (7h4m51.202510384s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615514677 (7h4m51.202570918s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615688160 (7h4m51.202744381s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615932236 (7h4m51.202988457s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616184166 (7h4m51.203240378s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616521716 (7h4m51.203577937s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616643614 (7h4m51.203699855s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621231117 (7h4m51.208287338s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621561163 (7h4m51.208617374s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621861323 (7h4m51.208917534s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.622156223 (7h4m51.209212444s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.726040884 (7h4m51.313097094s) [upload=7262]: Next returned nil segment Aug 7 23:31:37.726066772 (7h4m51.313122983s) [upload=7262]: Waiting for error group managing segments... Aug 7 23:31:37.736232517 (7h4m51.323288738s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.737656014 (7h4m51.324712245s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.739754932 (7h4m51.326811153s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.740628113 (7h4m51.327684344s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741712657 (7h4m51.328768867s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741742653 (7h4m51.328798883s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.748242274 (7h4m51.335298505s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.750434577 (7h4m51.337490798s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.758962364 (7h4m51.346018586s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.769693706 (7h4m51.356749917s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.800540734 (7h4m51.387596975s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.924012655 (7h4m51.511068866s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.059695219 (7h4m51.64675144s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102515917 (7h4m51.689572138s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102541606 (7h4m51.689597827s) [seg_pos={0 0}, upload=7262]: Segment reached optimal threshold of 14 pieces. Aug 7 23:31:38.102658574 (7h4m51.689714795s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF Aug 7 23:31:38.102748111 (7h4m51.689804332s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE): protocol: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection Aug 7 23:31:38.102832158 (7h4m51.689888399s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GcyWwKkfavsCsmtq55U72n6XeZKqQiGGYHY9D6UwST8sC14Ac): protocol: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection Aug 7 23:31:38.102967200 (7h4m51.690023421s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12syasUPsyYf4UstQ6cFAr6qamze3vCyAm9dnobk5e83w4bTz69): protocol: context canceled; context canceled Aug 7 23:31:38.102995132 (7h4m51.690051374s) [seg_pos={0 0}, upload=7262]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:31:38.103097553 (7h4m51.690153775s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:31:38.103102412 (7h4m51.690158633s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:31:38.203633212 (7h4m51.790689433s) [upload=7262]: Done uploading segments... Aug 7 23:31:38.203649102 (7h4m51.790705323s) [upload=7262]: Upload finished. err: <nil> Duration: 1.090935279s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [...XXXX.X..X..X......................................XXX.X.X] duration: 30.001332937s FAIL (2025-08-07 23:55:37 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001353746s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [....X.XXXXX..X..............................................] duration: 372.416746ms FAIL (2025-08-07 23:55:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 372.711828ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........XXX.X.X.........................................XX..] duration: 1.365937922s FAIL (2025-08-07 23:55:29 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ading piece... Aug 7 23:51:17.347399839 (7h24m30.934456051s) [noise=true , address=104.36.82.35:4611 , piece_id=FE47XNPAFZAELTDVIO6C4URDMXRWEXCM37BFM4LANQQ6QDSU7HAQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347417763 (7h24m30.934473974s) [noise=true , address=104.36.82.4:4806 , piece_id=X42UHVJVQMP46Y7QDE377JH6FNETULCU4JBDSGTVJDRSUE5LAVOA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347625520 (7h24m30.934681731s) [noise=true , address=45.91.116.10:4098 , piece_id=5ITNSAGAAD2QAEAZGQSRYU2GSIS54FPDHU6HETL6QVR57KUDYICQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347755754 (7h24m30.934811985s) [noise=true , address=104.36.82.45:4921 , piece_id=GQXCWGC6EHWZONLT426WVNPLASR7SECUANPQGGZNMASCRRD4X4SQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347968761 (7h24m30.935024982s) [noise=true , address=104.36.82.62:4438 , piece_id=3ND3RVOIIEJMN5GH7KZRHPYVNIH5KB62OU6KEV5DVOLCAZI4BQLA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348133560 (7h24m30.935189781s) [noise=true , address=104.36.82.39:4729 , piece_id=B7IGTQUWZAQNQDUYOUAB3WGYK5QHKXXJKRDGKMEGDK6KS4ZFC5GA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348193100 (7h24m30.935249311s) [noise=true , address=104.36.82.23:4475 , piece_id=LZMIGPR677ZHFYTXZJAMLW3GC3XLON7LGPRRBWZWVEXG4GGLEEEA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348536501 (7h24m30.935592722s) [noise=true , address=104.36.82.61:4406 , piece_id=N63ZWSHAILE55ZOUL5H6RYDUTB5F2F3EWMDT52RPJ5F46MLKRDMA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348656385 (7h24m30.935712596s) [noise=true , address=104.36.82.12:4178 , piece_id=NQWZGYOLKX6SHV3RXIXVGFDAXFP4VMFMP5FJKQPZ63YZ2NK3RJDQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348854374 (7h24m30.935910585s) [noise=true , address=104.36.82.15:4288 , piece_id=KWW7E4ZMO42C3YEAH3WBCEYSW6TTNDX7TQ4MEITLSC2RR2CE4VDA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348919236 (7h24m30.935975457s) [noise=true , address=104.36.82.36:4649 , piece_id=NV2V22M4HHIEHH4RENXMFHFYXHQF7LPJRPPD7ZBKHPSZFKI2BVTA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.430583697 (7h24m31.017639918s) [upload=7600]: Next returned nil segment Aug 7 23:51:17.430596350 (7h24m31.017652571s) [upload=7600]: Waiting for error group managing segments... Aug 7 23:51:17.450496095 (7h24m31.037552306s) [noise=true , address=104.36.82.39:4729 , piece_id=B7IGTQUWZAQNQDUYOUAB3WGYK5QHKXXJKRDGKMEGDK6KS4ZFC5GA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.457273115 (7h24m31.044329336s) [noise=true , address=45.91.116.11:4121 , piece_id=UI4HOUG4KLWC3LFGDVZ3C3IUMB2DGJP2QOTIOLWS5C2HUAMSATMA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.458658320 (7h24m31.045714532s) [noise=true , address=104.36.82.36:4649 , piece_id=NV2V22M4HHIEHH4RENXMFHFYXHQF7LPJRPPD7ZBKHPSZFKI2BVTA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.464324767 (7h24m31.051380978s) [noise=true , address=104.36.82.62:4438 , piece_id=3ND3RVOIIEJMN5GH7KZRHPYVNIH5KB62OU6KEV5DVOLCAZI4BQLA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.469258046 (7h24m31.056314256s) [noise=true , address=104.36.82.4:4806 , piece_id=X42UHVJVQMP46Y7QDE377JH6FNETULCU4JBDSGTVJDRSUE5LAVOA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.470547041 (7h24m31.057603252s) [noise=true , address=45.91.116.10:4098 , piece_id=5ITNSAGAAD2QAEAZGQSRYU2GSIS54FPDHU6HETL6QVR57KUDYICQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.482895480 (7h24m31.069951691s) [noise=true , address=104.36.82.59:4327 , piece_id=5AM7TVDV5ORD5WPHKEHOM652JETCJ6VH3JOJ6HPJTIOTABTYVELA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.503848269 (7h24m31.090904481s) [noise=true , address=104.36.82.61:4406 , piece_id=N63ZWSHAILE55ZOUL5H6RYDUTB5F2F3EWMDT52RPJ5F46MLKRDMA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.589438619 (7h24m31.17649484s) [noise=true , address=79.127.201.215:4343 , piece_id=6IVWBNSJNI4ZAEIWFTRYUHIHZ35YTTZF3FTEF7G76YEBUP2NBJFA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.633863581 (7h24m31.220919802s) [noise=true , address=104.36.82.42:4801 , piece_id=ILNDV4CZCPEST6R4DUTSHQ5OJS2N4BLPRLDYIYNBTZKRR4RSCOFA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.636493760 (7h24m31.22354997s) [noise=true , address=104.36.82.44:4904 , piece_id=TFXFZ53DGPOVVXTTZMRMTJR53DGXNHWUF43I7OCC4YNCRW4QUCKA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.637613389 (7h24m31.224669611s) [noise=true , address=104.36.82.23:4475 , piece_id=LZMIGPR677ZHFYTXZJAMLW3GC3XLON7LGPRRBWZWVEXG4GGLEEEA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.654845844 (7h24m31.241902065s) [noise=true , address=104.36.82.22:4524 , piece_id=66YIUJKSDV52T5NMMFHYCZH7L34UKH3RLIS5WZ3VWRBL3KEE7UWQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.853873712 (7h24m31.440929933s) [noise=true , address=104.36.82.12:4178 , piece_id=NQWZGYOLKX6SHV3RXIXVGFDAXFP4VMFMP5FJKQPZ63YZ2NK3RJDQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.853888850 (7h24m31.440945061s) [seg_pos={0 0}, upload=7600]: Segment reached optimal threshold of 14 pieces. Aug 7 23:51:17.853987013 (7h24m31.441043234s) [noise=true , address=104.36.82.45:4921 , piece_id=GQXCWGC6EHWZONLT426WVNPLASR7SECUANPQGGZNMASCRRD4X4SQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bBRKXiuSN2cw6KrdP8q4Dsf4MiFCUSgMEGjqG2dKL9RLTQra1): protocol: expected piece hash; context canceled; EOF Aug 7 23:51:17.854295209 (7h24m31.44135142s) [noise=true , address=104.36.82.46:4499 , piece_id=6OBTEQJKT4AJRGUT27EYWNLG4E2TKBCQBRGXTTEOU4SJY2SWUTFQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc): protocol: write tcp 5.161.89.252:63506->104.36.82.46:4499: use of closed network connection; write tcp 5.161.89.252:63506->104.36.82.46:4499: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63506->104.36.82.46:4499: use of closed network connection Aug 7 23:51:17.854338279 (7h24m31.44139449s) [noise=true , address=104.36.82.35:4611 , piece_id=FE47XNPAFZAELTDVIO6C4URDMXRWEXCM37BFM4LANQQ6QDSU7HAQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zkqEMSKx7zSHPFNZ51b5TLzL2cFQL8VyxAG14Lg1pT8zAu3tc): protocol: write tcp 5.161.89.252:46266->104.36.82.35:4611: use of closed network connection; write tcp 5.161.89.252:46266->104.36.82.35:4611: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:46266->104.36.82.35:4611: use of closed network connection Aug 7 23:51:17.854399554 (7h24m31.441455765s) [noise=true , address=104.36.82.15:4288 , piece_id=KWW7E4ZMO42C3YEAH3WBCEYSW6TTNDX7TQ4MEITLSC2RR2CE4VDA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gPjPGuueZnizLT2QvJN9y7riwKiW5TWMkQtmGGPP7VZvkNoB7): protocol: read tcp 5.161.89.252:35428->104.36.82.15:4288: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:51:17.854419941 (7h24m31.441476162s) [seg_pos={0 0}, upload=7600]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:51:17.854472530 (7h24m31.441528751s) [upload=7600]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:51:17.854476738 (7h24m31.441532959s) [upload=7600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:17.900817215 (7h24m31.487873465s) [upload=7600]: Done uploading segments... Aug 7 23:51:17.900834827 (7h24m31.487891079s) [upload=7600]: Upload finished. err: <nil> Aug 7 23:52:11.382772324 (7h25m24.969828534s) [upload=7616]: Starting upload Aug 7 23:52:11.382828459 (7h25m24.96988468s) [upload=7616]: Done starting upload Aug 7 23:52:11.382871529 (7h25m24.96992774s) [upload=7616]: Uploading segments... Aug 7 23:52:11.383053639 (7h25m24.97010986s) [upload=7616]: Got next segment. Inline: false Aug 7 23:52:11.383080680 (7h25m24.970136891s) [upload=7616]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:11.383084406 (7h25m24.970140617s) [upload=7616]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:11.371970391 (7h26m24.959026631s) [upload=7616]: Next segment err: context deadline exceeded Aug 7 23:53:11.372056972 (7h26m24.959113223s) [upload=7616]: Done uploading segments... Aug 7 23:53:11.372090856 (7h26m24.959147096s) [upload=7616]: Upload finished. err: context deadline exceeded Aug 7 23:53:11.462534524 (7h26m25.049590775s) [upload=7633]: Starting upload Aug 7 23:53:11.462662182 (7h26m25.049718423s) [upload=7633]: Done starting upload Aug 7 23:53:11.462730148 (7h26m25.0497864s) [upload=7633]: Uploading segments... Aug 7 23:53:11.463218160 (7h26m25.050274451s) [upload=7633]: Got next segment. Inline: false Aug 7 23:53:11.463267292 (7h26m25.050323533s) [upload=7633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:11.463276689 (7h26m25.050332931s) [upload=7633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:54:11.448954917 (7h27m25.036011128s) [upload=7633]: Next segment err: context deadline exceeded Aug 7 23:54:11.449030117 (7h27m25.036086328s) [upload=7633]: Done uploading segments... Aug 7 23:54:11.449047229 (7h27m25.03610344s) [upload=7633]: Upload finished. err: context deadline exceeded Duration: 1.365959312s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 9 out of 60 is failed (threshold=3) [..XXXXXX.X...............................................XX.] duration: 228.131792ms FAIL (2025-08-07 23:55:12 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 95s) [upload=7512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:05.713026270 (7h19m19.300082522s) [upload=7512]: Done uploading segments... Aug 7 23:46:05.713042230 (7h19m19.300098472s) [upload=7512]: Upload finished. err: <nil> Aug 7 23:46:59.827069649 (7h20m13.41412587s) [upload=7526]: Starting upload Aug 7 23:46:59.827112839 (7h20m13.414169051s) [upload=7526]: Done starting upload Aug 7 23:46:59.827158284 (7h20m13.414214506s) [upload=7526]: Uploading segments... Aug 7 23:46:59.827184925 (7h20m13.414241135s) [upload=7526]: Got next segment. Inline: true Aug 7 23:46:59.827188771 (7h20m13.414244992s) [upload=7526]: Waiting for error group managing segments... Aug 7 23:46:59.827219097 (7h20m13.414275308s) [upload=7526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:59.827222835 (7h20m13.414279056s) [upload=7526]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:59.827225119 (7h20m13.414281329s) [upload=7526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:59.879507639 (7h20m13.46656385s) [upload=7526]: Done uploading segments... Aug 7 23:46:59.879523058 (7h20m13.466579269s) [upload=7526]: Upload finished. err: <nil> Aug 7 23:47:39.924312697 (7h20m53.511368928s) [upload=7536]: Starting upload Aug 7 23:47:39.924413405 (7h20m53.511469646s) [upload=7536]: Done starting upload Aug 7 23:47:39.924491079 (7h20m53.51154731s) [upload=7536]: Uploading segments... Aug 7 23:47:39.924700320 (7h20m53.511756551s) [upload=7536]: Got next segment. Inline: true Aug 7 23:47:39.924711842 (7h20m53.511768063s) [upload=7536]: Waiting for error group managing segments... Aug 7 23:47:39.924828248 (7h20m53.51188448s) [upload=7536]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:47:39.924836915 (7h20m53.511893156s) [upload=7536]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:47:39.924842024 (7h20m53.511898255s) [upload=7536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:39.988158721 (7h20m53.575214932s) [upload=7536]: Done uploading segments... Aug 7 23:47:39.988176785 (7h20m53.575232995s) [upload=7536]: Upload finished. err: <nil> Aug 7 23:48:17.585904864 (7h21m31.172961076s) [upload=7548]: Starting upload Aug 7 23:48:17.585990855 (7h21m31.173047066s) [upload=7548]: Done starting upload Aug 7 23:48:17.586065835 (7h21m31.173122035s) [upload=7548]: Uploading segments... Aug 7 23:48:17.586102964 (7h21m31.173159205s) [upload=7548]: Got next segment. Inline: true Aug 7 23:48:17.586107262 (7h21m31.173163473s) [upload=7548]: Waiting for error group managing segments... Aug 7 23:48:17.586144451 (7h21m31.173200662s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:17.586148969 (7h21m31.17320518s) [upload=7548]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:17.586151854 (7h21m31.173208066s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:17.642355873 (7h21m31.229412094s) [upload=7548]: Done uploading segments... Aug 7 23:48:17.642371582 (7h21m31.229427794s) [upload=7548]: Upload finished. err: <nil> Aug 7 23:49:19.929340015 (7h22m33.516396255s) [upload=7565]: Starting upload Aug 7 23:49:19.929497669 (7h22m33.51655391s) [upload=7565]: Done starting upload Aug 7 23:49:19.929593468 (7h22m33.516649709s) [upload=7565]: Uploading segments... Aug 7 23:49:19.929640947 (7h22m33.516697188s) [upload=7565]: Got next segment. Inline: true Aug 7 23:49:19.929647679 (7h22m33.516703931s) [upload=7565]: Waiting for error group managing segments... Aug 7 23:49:19.929695078 (7h22m33.516751329s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:19.929701059 (7h22m33.5167573s) [upload=7565]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:19.929704956 (7h22m33.516761197s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:19.992636681 (7h22m33.579692922s) [upload=7565]: Done uploading segments... Aug 7 23:49:19.992654385 (7h22m33.579710625s) [upload=7565]: Upload finished. err: <nil> Aug 7 23:50:07.176468149 (7h23m20.76352438s) [upload=7576]: Starting upload Aug 7 23:50:07.176537798 (7h23m20.763594019s) [upload=7576]: Done starting upload Aug 7 23:50:07.176592070 (7h23m20.763648311s) [upload=7576]: Uploading segments... Aug 7 23:50:07.176626473 (7h23m20.763682705s) [upload=7576]: Got next segment. Inline: true Aug 7 23:50:07.176631002 (7h23m20.763687213s) [upload=7576]: Waiting for error group managing segments... Aug 7 23:50:07.176664084 (7h23m20.763720295s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:07.176668132 (7h23m20.763724343s) [upload=7576]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:07.176670586 (7h23m20.763726807s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:07.240851653 (7h23m20.827907864s) [upload=7576]: Done uploading segments... Aug 7 23:50:07.240866771 (7h23m20.827922992s) [upload=7576]: Upload finished. err: <nil> Aug 7 23:50:45.893754262 (7h23m59.480810504s) [upload=7591]: Starting upload Aug 7 23:50:45.893825847 (7h23m59.480882088s) [upload=7591]: Done starting upload Aug 7 23:50:45.893912749 (7h23m59.48096899s) [upload=7591]: Uploading segments... Aug 7 23:50:45.893955949 (7h23m59.4810122s) [upload=7591]: Got next segment. Inline: true Aug 7 23:50:45.893961970 (7h23m59.481018211s) [upload=7591]: Waiting for error group managing segments... Aug 7 23:50:45.894005482 (7h23m59.481061723s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:45.894011062 (7h23m59.481067304s) [upload=7591]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:45.894014729 (7h23m59.48107097s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:05.079621048 (7h24m18.666677259s) [upload=7591]: Done uploading segments... Aug 7 23:51:05.079636297 (7h24m18.666692508s) [upload=7591]: Upload finished. err: <nil> Aug 7 23:51:21.725622238 (7h24m35.312678469s) [upload=7603]: Starting upload Aug 7 23:51:21.725696627 (7h24m35.312752838s) [upload=7603]: Done starting upload Aug 7 23:51:21.725758231 (7h24m35.312814443s) [upload=7603]: Uploading segments... Aug 7 23:51:21.725796774 (7h24m35.312852985s) [upload=7603]: Got next segment. Inline: true Aug 7 23:51:21.725802845 (7h24m35.312859056s) [upload=7603]: Waiting for error group managing segments... Aug 7 23:51:21.725848871 (7h24m35.312905082s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:21.725855313 (7h24m35.312911523s) [upload=7603]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:21.725859100 (7h24m35.312915351s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:21.784505948 (7h24m35.371562158s) [upload=7603]: Done uploading segments... Aug 7 23:51:21.784521767 (7h24m35.371577978s) [upload=7603]: Upload finished. err: <nil> Aug 7 23:52:10.285208176 (7h25m23.872264397s) [upload=7615]: Starting upload Aug 7 23:52:10.285263228 (7h25m23.872319449s) [upload=7615]: Done starting upload Aug 7 23:52:10.285310727 (7h25m23.872366937s) [upload=7615]: Uploading segments... Aug 7 23:52:10.285339711 (7h25m23.872395932s) [upload=7615]: Got next segment. Inline: true Aug 7 23:52:10.285343438 (7h25m23.872399649s) [upload=7615]: Waiting for error group managing segments... Aug 7 23:52:10.285391158 (7h25m23.872447379s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:10.285395065 (7h25m23.872451276s) [upload=7615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:10.285397770 (7h25m23.872453991s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:10.400888570 (7h25m23.987944781s) [upload=7615]: Done uploading segments... Aug 7 23:52:10.400904530 (7h25m23.987960741s) [upload=7615]: Upload finished. err: <nil> Aug 7 23:53:11.795016020 (7h26m25.382072262s) [upload=7635]: Starting upload Aug 7 23:53:11.795108102 (7h26m25.382164354s) [upload=7635]: Done starting upload Aug 7 23:53:11.795175588 (7h26m25.382231829s) [upload=7635]: Uploading segments... Aug 7 23:53:11.795254645 (7h26m25.382310886s) [upload=7635]: Got next segment. Inline: true Aug 7 23:53:11.795262130 (7h26m25.38231837s) [upload=7635]: Waiting for error group managing segments... Aug 7 23:53:11.795319296 (7h26m25.382375538s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:11.795326170 (7h26m25.38238242s) [upload=7635]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:11.795329857 (7h26m25.382386097s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:11.782974743 (7h27m25.370030954s) [upload=7635]: Done uploading segments... Aug 7 23:54:11.782991474 (7h27m25.370047695s) [upload=7635]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:11.855031501 (7h27m25.442087722s) [upload=7650]: Starting upload Aug 7 23:54:11.855087627 (7h27m25.442143838s) [upload=7650]: Done starting upload Aug 7 23:54:11.855131608 (7h27m25.442187829s) [upload=7650]: Uploading segments... Aug 7 23:54:11.855163227 (7h27m25.442219448s) [upload=7650]: Got next segment. Inline: true Aug 7 23:54:11.855167005 (7h27m25.442223226s) [upload=7650]: Waiting for error group managing segments... Aug 7 23:54:11.855198905 (7h27m25.442255126s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:11.855202661 (7h27m25.442258882s) [upload=7650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:11.855205036 (7h27m25.442261247s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:11.844108363 (7h28m25.431164634s) [upload=7650]: Done uploading segments... Aug 7 23:55:11.844129542 (7h28m25.431185763s) [upload=7650]: Upload finished. err: metaclient: context deadline exceeded Duration:* 228.147561ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 9 out of 60 is failed (threshold=3) [...XXXXXX.X...............................................XX] duration: 1m0.001690304s FAIL (2025-08-07 23:55:11 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 95s) [upload=7512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:05.713026270 (7h19m19.300082522s) [upload=7512]: Done uploading segments... Aug 7 23:46:05.713042230 (7h19m19.300098472s) [upload=7512]: Upload finished. err: <nil> Aug 7 23:46:59.827069649 (7h20m13.41412587s) [upload=7526]: Starting upload Aug 7 23:46:59.827112839 (7h20m13.414169051s) [upload=7526]: Done starting upload Aug 7 23:46:59.827158284 (7h20m13.414214506s) [upload=7526]: Uploading segments... Aug 7 23:46:59.827184925 (7h20m13.414241135s) [upload=7526]: Got next segment. Inline: true Aug 7 23:46:59.827188771 (7h20m13.414244992s) [upload=7526]: Waiting for error group managing segments... Aug 7 23:46:59.827219097 (7h20m13.414275308s) [upload=7526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:59.827222835 (7h20m13.414279056s) [upload=7526]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:59.827225119 (7h20m13.414281329s) [upload=7526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:59.879507639 (7h20m13.46656385s) [upload=7526]: Done uploading segments... Aug 7 23:46:59.879523058 (7h20m13.466579269s) [upload=7526]: Upload finished. err: <nil> Aug 7 23:47:39.924312697 (7h20m53.511368928s) [upload=7536]: Starting upload Aug 7 23:47:39.924413405 (7h20m53.511469646s) [upload=7536]: Done starting upload Aug 7 23:47:39.924491079 (7h20m53.51154731s) [upload=7536]: Uploading segments... Aug 7 23:47:39.924700320 (7h20m53.511756551s) [upload=7536]: Got next segment. Inline: true Aug 7 23:47:39.924711842 (7h20m53.511768063s) [upload=7536]: Waiting for error group managing segments... Aug 7 23:47:39.924828248 (7h20m53.51188448s) [upload=7536]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:47:39.924836915 (7h20m53.511893156s) [upload=7536]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:47:39.924842024 (7h20m53.511898255s) [upload=7536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:39.988158721 (7h20m53.575214932s) [upload=7536]: Done uploading segments... Aug 7 23:47:39.988176785 (7h20m53.575232995s) [upload=7536]: Upload finished. err: <nil> Aug 7 23:48:17.585904864 (7h21m31.172961076s) [upload=7548]: Starting upload Aug 7 23:48:17.585990855 (7h21m31.173047066s) [upload=7548]: Done starting upload Aug 7 23:48:17.586065835 (7h21m31.173122035s) [upload=7548]: Uploading segments... Aug 7 23:48:17.586102964 (7h21m31.173159205s) [upload=7548]: Got next segment. Inline: true Aug 7 23:48:17.586107262 (7h21m31.173163473s) [upload=7548]: Waiting for error group managing segments... Aug 7 23:48:17.586144451 (7h21m31.173200662s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:17.586148969 (7h21m31.17320518s) [upload=7548]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:17.586151854 (7h21m31.173208066s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:17.642355873 (7h21m31.229412094s) [upload=7548]: Done uploading segments... Aug 7 23:48:17.642371582 (7h21m31.229427794s) [upload=7548]: Upload finished. err: <nil> Aug 7 23:49:19.929340015 (7h22m33.516396255s) [upload=7565]: Starting upload Aug 7 23:49:19.929497669 (7h22m33.51655391s) [upload=7565]: Done starting upload Aug 7 23:49:19.929593468 (7h22m33.516649709s) [upload=7565]: Uploading segments... Aug 7 23:49:19.929640947 (7h22m33.516697188s) [upload=7565]: Got next segment. Inline: true Aug 7 23:49:19.929647679 (7h22m33.516703931s) [upload=7565]: Waiting for error group managing segments... Aug 7 23:49:19.929695078 (7h22m33.516751329s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:19.929701059 (7h22m33.5167573s) [upload=7565]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:19.929704956 (7h22m33.516761197s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:19.992636681 (7h22m33.579692922s) [upload=7565]: Done uploading segments... Aug 7 23:49:19.992654385 (7h22m33.579710625s) [upload=7565]: Upload finished. err: <nil> Aug 7 23:50:07.176468149 (7h23m20.76352438s) [upload=7576]: Starting upload Aug 7 23:50:07.176537798 (7h23m20.763594019s) [upload=7576]: Done starting upload Aug 7 23:50:07.176592070 (7h23m20.763648311s) [upload=7576]: Uploading segments... Aug 7 23:50:07.176626473 (7h23m20.763682705s) [upload=7576]: Got next segment. Inline: true Aug 7 23:50:07.176631002 (7h23m20.763687213s) [upload=7576]: Waiting for error group managing segments... Aug 7 23:50:07.176664084 (7h23m20.763720295s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:07.176668132 (7h23m20.763724343s) [upload=7576]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:07.176670586 (7h23m20.763726807s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:07.240851653 (7h23m20.827907864s) [upload=7576]: Done uploading segments... Aug 7 23:50:07.240866771 (7h23m20.827922992s) [upload=7576]: Upload finished. err: <nil> Aug 7 23:50:45.893754262 (7h23m59.480810504s) [upload=7591]: Starting upload Aug 7 23:50:45.893825847 (7h23m59.480882088s) [upload=7591]: Done starting upload Aug 7 23:50:45.893912749 (7h23m59.48096899s) [upload=7591]: Uploading segments... Aug 7 23:50:45.893955949 (7h23m59.4810122s) [upload=7591]: Got next segment. Inline: true Aug 7 23:50:45.893961970 (7h23m59.481018211s) [upload=7591]: Waiting for error group managing segments... Aug 7 23:50:45.894005482 (7h23m59.481061723s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:45.894011062 (7h23m59.481067304s) [upload=7591]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:45.894014729 (7h23m59.48107097s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:05.079621048 (7h24m18.666677259s) [upload=7591]: Done uploading segments... Aug 7 23:51:05.079636297 (7h24m18.666692508s) [upload=7591]: Upload finished. err: <nil> Aug 7 23:51:21.725622238 (7h24m35.312678469s) [upload=7603]: Starting upload Aug 7 23:51:21.725696627 (7h24m35.312752838s) [upload=7603]: Done starting upload Aug 7 23:51:21.725758231 (7h24m35.312814443s) [upload=7603]: Uploading segments... Aug 7 23:51:21.725796774 (7h24m35.312852985s) [upload=7603]: Got next segment. Inline: true Aug 7 23:51:21.725802845 (7h24m35.312859056s) [upload=7603]: Waiting for error group managing segments... Aug 7 23:51:21.725848871 (7h24m35.312905082s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:21.725855313 (7h24m35.312911523s) [upload=7603]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:21.725859100 (7h24m35.312915351s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:21.784505948 (7h24m35.371562158s) [upload=7603]: Done uploading segments... Aug 7 23:51:21.784521767 (7h24m35.371577978s) [upload=7603]: Upload finished. err: <nil> Aug 7 23:52:10.285208176 (7h25m23.872264397s) [upload=7615]: Starting upload Aug 7 23:52:10.285263228 (7h25m23.872319449s) [upload=7615]: Done starting upload Aug 7 23:52:10.285310727 (7h25m23.872366937s) [upload=7615]: Uploading segments... Aug 7 23:52:10.285339711 (7h25m23.872395932s) [upload=7615]: Got next segment. Inline: true Aug 7 23:52:10.285343438 (7h25m23.872399649s) [upload=7615]: Waiting for error group managing segments... Aug 7 23:52:10.285391158 (7h25m23.872447379s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:10.285395065 (7h25m23.872451276s) [upload=7615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:10.285397770 (7h25m23.872453991s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:10.400888570 (7h25m23.987944781s) [upload=7615]: Done uploading segments... Aug 7 23:52:10.400904530 (7h25m23.987960741s) [upload=7615]: Upload finished. err: <nil> Aug 7 23:53:11.795016020 (7h26m25.382072262s) [upload=7635]: Starting upload Aug 7 23:53:11.795108102 (7h26m25.382164354s) [upload=7635]: Done starting upload Aug 7 23:53:11.795175588 (7h26m25.382231829s) [upload=7635]: Uploading segments... Aug 7 23:53:11.795254645 (7h26m25.382310886s) [upload=7635]: Got next segment. Inline: true Aug 7 23:53:11.795262130 (7h26m25.38231837s) [upload=7635]: Waiting for error group managing segments... Aug 7 23:53:11.795319296 (7h26m25.382375538s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:11.795326170 (7h26m25.38238242s) [upload=7635]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:11.795329857 (7h26m25.382386097s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:11.782974743 (7h27m25.370030954s) [upload=7635]: Done uploading segments... Aug 7 23:54:11.782991474 (7h27m25.370047695s) [upload=7635]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:11.855031501 (7h27m25.442087722s) [upload=7650]: Starting upload Aug 7 23:54:11.855087627 (7h27m25.442143838s) [upload=7650]: Done starting upload Aug 7 23:54:11.855131608 (7h27m25.442187829s) [upload=7650]: Uploading segments... Aug 7 23:54:11.855163227 (7h27m25.442219448s) [upload=7650]: Got next segment. Inline: true Aug 7 23:54:11.855167005 (7h27m25.442223226s) [upload=7650]: Waiting for error group managing segments... Aug 7 23:54:11.855198905 (7h27m25.442255126s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:11.855202661 (7h27m25.442258882s) [upload=7650]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:11.855205036 (7h27m25.442261247s) [upload=7650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:11.844108363 (7h28m25.431164634s) [upload=7650]: Done uploading segments... Aug 7 23:55:11.844129542 (7h28m25.431185763s) [upload=7650]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1m0.001703699s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.XX.X....X.X..........................................X...] duration: 1.026320259s FAIL (2025-08-07 23:55:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.026338653s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 13 out of 60 is failed (threshold=3) [..XXXXX.XX............................................XXXXXX] duration: 30.000769403s FAIL (2025-08-07 23:55:01 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 10 (7h20m13.662582431s) [upload=7527]: Done uploading segments... Aug 7 23:47:00.075540917 (7h20m13.662597138s) [upload=7527]: Upload finished. err: <nil> Aug 7 23:48:09.882765252 (7h21m23.469821463s) [upload=7545]: Starting upload Aug 7 23:48:09.882860198 (7h21m23.46991642s) [upload=7545]: Done starting upload Aug 7 23:48:09.882935729 (7h21m23.469991941s) [upload=7545]: Uploading segments... Aug 7 23:48:09.882965635 (7h21m23.470021856s) [upload=7545]: Got next segment. Inline: true Aug 7 23:48:09.882969723 (7h21m23.470025944s) [upload=7545]: Waiting for error group managing segments... Aug 7 23:48:09.883002865 (7h21m23.470059086s) [upload=7545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:09.883006481 (7h21m23.470062702s) [upload=7545]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:09.883009045 (7h21m23.470065266s) [upload=7545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:09.951272229 (7h21m23.538328451s) [upload=7545]: Done uploading segments... Aug 7 23:48:09.951286926 (7h21m23.538343148s) [upload=7545]: Upload finished. err: <nil> Aug 7 23:48:58.772502770 (7h22m12.359559011s) [upload=7557]: Starting upload Aug 7 23:48:58.772574805 (7h22m12.359631046s) [upload=7557]: Done starting upload Aug 7 23:48:58.772645527 (7h22m12.359701767s) [upload=7557]: Uploading segments... Aug 7 23:48:58.772695300 (7h22m12.359751541s) [upload=7557]: Got next segment. Inline: true Aug 7 23:48:58.772701031 (7h22m12.359757271s) [upload=7557]: Waiting for error group managing segments... Aug 7 23:48:58.772770249 (7h22m12.359826501s) [upload=7557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:58.772780228 (7h22m12.359836469s) [upload=7557]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:58.772787010 (7h22m12.359843252s) [upload=7557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:58.840436652 (7h22m12.427492893s) [upload=7557]: Done uploading segments... Aug 7 23:48:58.840462661 (7h22m12.427518891s) [upload=7557]: Upload finished. err: <nil> Aug 7 23:50:01.971851413 (7h23m15.558907624s) [upload=7574]: Starting upload Aug 7 23:50:01.971890696 (7h23m15.558946907s) [upload=7574]: Done starting upload Aug 7 23:50:01.971977749 (7h23m15.559033961s) [upload=7574]: Uploading segments... Aug 7 23:50:01.972022943 (7h23m15.559079154s) [upload=7574]: Got next segment. Inline: true Aug 7 23:50:01.972026850 (7h23m15.559083062s) [upload=7574]: Waiting for error group managing segments... Aug 7 23:50:01.972060975 (7h23m15.559117185s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:01.972064511 (7h23m15.559120712s) [upload=7574]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:01.972066685 (7h23m15.559122896s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:02.063709030 (7h23m15.650765241s) [upload=7574]: Done uploading segments... Aug 7 23:50:02.063723547 (7h23m15.650779768s) [upload=7574]: Upload finished. err: <nil> Aug 7 23:50:41.160166556 (7h23m54.747222777s) [upload=7588]: Starting upload Aug 7 23:50:41.160220555 (7h23m54.747276777s) [upload=7588]: Done starting upload Aug 7 23:50:41.160264659 (7h23m54.74732088s) [upload=7588]: Uploading segments... Aug 7 23:50:41.160295956 (7h23m54.747352177s) [upload=7588]: Got next segment. Inline: true Aug 7 23:50:41.160299753 (7h23m54.747355964s) [upload=7588]: Waiting for error group managing segments... Aug 7 23:50:41.160331633 (7h23m54.747387844s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:41.160335861 (7h23m54.747392072s) [upload=7588]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:41.160338396 (7h23m54.747394617s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:41.222859735 (7h23m54.809915976s) [upload=7588]: Done uploading segments... Aug 7 23:50:41.222879913 (7h23m54.809936154s) [upload=7588]: Upload finished. err: <nil> Aug 7 23:51:17.131690986 (7h24m30.718747197s) [upload=7599]: Starting upload Aug 7 23:51:17.131738926 (7h24m30.718795147s) [upload=7599]: Done starting upload Aug 7 23:51:17.131792947 (7h24m30.718849178s) [upload=7599]: Uploading segments... Aug 7 23:51:17.131835896 (7h24m30.718892117s) [upload=7599]: Got next segment. Inline: true Aug 7 23:51:17.131842027 (7h24m30.718898239s) [upload=7599]: Waiting for error group managing segments... Aug 7 23:51:17.131883385 (7h24m30.718939627s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:17.131889146 (7h24m30.718945388s) [upload=7599]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:17.132006205 (7h24m30.719062435s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:38.787433326 (7h24m52.374489567s) [upload=7599]: Done uploading segments... Aug 7 23:51:38.787452212 (7h24m52.374508453s) [upload=7599]: Upload finished. err: <nil> Aug 7 23:52:00.256204647 (7h25m13.843260858s) [upload=7610]: Starting upload Aug 7 23:52:00.256263547 (7h25m13.843319758s) [upload=7610]: Done starting upload Aug 7 23:52:00.256316195 (7h25m13.843372406s) [upload=7610]: Uploading segments... Aug 7 23:52:00.256345671 (7h25m13.843401881s) [upload=7610]: Got next segment. Inline: true Aug 7 23:52:00.256349758 (7h25m13.843405969s) [upload=7610]: Waiting for error group managing segments... Aug 7 23:52:00.256399992 (7h25m13.843456203s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:00.256404780 (7h25m13.843460991s) [upload=7610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:00.256407806 (7h25m13.843464017s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:30.237600164 (7h25m43.824656416s) [upload=7610]: Done uploading segments... Aug 7 23:52:30.237630862 (7h25m43.824687113s) [upload=7610]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:53.146251246 (7h26m6.733307456s) [upload=7626]: Starting upload Aug 7 23:52:53.146307339 (7h26m6.733363561s) [upload=7626]: Done starting upload Aug 7 23:52:53.146350962 (7h26m6.733407173s) [upload=7626]: Uploading segments... Aug 7 23:52:53.146411344 (7h26m6.733467565s) [upload=7626]: Got next segment. Inline: true Aug 7 23:52:53.146415271 (7h26m6.733471493s) [upload=7626]: Waiting for error group managing segments... Aug 7 23:52:53.146449966 (7h26m6.733506177s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:53.146454174 (7h26m6.733510395s) [upload=7626]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:53.146456709 (7h26m6.73351293s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.132536521 (7h26m36.719592742s) [upload=7626]: Done uploading segments... Aug 7 23:53:23.132560907 (7h26m36.719617128s) [upload=7626]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:53:28.217132518 (7h26m41.804188759s) [upload=7638]: Starting upload Aug 7 23:53:28.217216185 (7h26m41.804272426s) [upload=7638]: Done starting upload Aug 7 23:53:28.217288009 (7h26m41.80434426s) [upload=7638]: Uploading segments... Aug 7 23:53:28.217336129 (7h26m41.80439237s) [upload=7638]: Got next segment. Inline: true Aug 7 23:53:28.217341890 (7h26m41.804398131s) [upload=7638]: Waiting for error group managing segments... Aug 7 23:53:28.217421438 (7h26m41.804477679s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:28.217427650 (7h26m41.804483901s) [upload=7638]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:28.217431617 (7h26m41.804487859s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:58.204830989 (7h27m11.79188721s) [upload=7638]: Done uploading segments... Aug 7 23:53:58.204849294 (7h27m11.791905505s) [upload=7638]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:01.364416813 (7h27m14.951473054s) [upload=7648]: Starting upload Aug 7 23:54:01.364544481 (7h27m14.951600732s) [upload=7648]: Done starting upload Aug 7 23:54:01.364639518 (7h27m14.95169577s) [upload=7648]: Uploading segments... Aug 7 23:54:01.364703858 (7h27m14.951760109s) [upload=7648]: Got next segment. Inline: true Aug 7 23:54:01.364713517 (7h27m14.951769757s) [upload=7648]: Waiting for error group managing segments... Aug 7 23:54:01.364781374 (7h27m14.951837625s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:01.364791231 (7h27m14.951847472s) [upload=7648]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:01.364797502 (7h27m14.951853754s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:31.351186626 (7h27m44.938242847s) [upload=7648]: Done uploading segments... Aug 7 23:54:31.351241118 (7h27m44.938297339s) [upload=7648]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:31.416140839 (7h27m45.003197049s) [upload=7656]: Starting upload Aug 7 23:54:31.416189309 (7h27m45.00324553s) [upload=7656]: Done starting upload Aug 7 23:54:31.416235946 (7h27m45.003292157s) [upload=7656]: Uploading segments... Aug 7 23:54:31.416277384 (7h27m45.003333595s) [upload=7656]: Got next segment. Inline: true Aug 7 23:54:31.416297601 (7h27m45.003353813s) [upload=7656]: Waiting for error group managing segments... Aug 7 23:54:31.416336454 (7h27m45.003392665s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:31.416340652 (7h27m45.003396873s) [upload=7656]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:31.416342965 (7h27m45.003399177s) [upload=7656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:01.405081824 (7h28m14.992138035s) [upload=7656]: Done uploading segments... Aug 7 23:55:01.405101451 (7h28m14.992157672s) [upload=7656]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000779852s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.....XXXXX..X............................................XXX] duration: 1m0.000920548s FAIL (2025-08-07 23:55:00 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000931317s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.....X.XXXXX..X.............................................] duration: 376.124477ms FAIL (2025-08-07 23:54:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 376.149834ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 133.66315ms FAIL (2025-08-07 23:54:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 133.723323ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..XXXXXX.X.X...........................................X.X..] duration: 1.740102805s FAIL (2025-08-07 23:54:48 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.740120539s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.041137846s FAIL (2025-08-07 23:54:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== 1345s) [upload=7262]: Got next segment. Inline: false Aug 7 23:31:37.583906637 (7h4m51.170962878s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:31:37.583953414 (7h4m51.171009656s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:31:37.613963041 (7h4m51.201019262s) [seg_pos={0 0}, upload=7262]: Begin upload segment... Aug 7 23:31:37.614008966 (7h4m51.201065176s) [seg_pos={0 0}, upload=7262]: Done begin upload segment. Aug 7 23:31:37.614029103 (7h4m51.201085324s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614108311 (7h4m51.201164542s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614344612 (7h4m51.201400833s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614634754 (7h4m51.201690975s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614919926 (7h4m51.201976137s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614943150 (7h4m51.201999391s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615146719 (7h4m51.20220294s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615454163 (7h4m51.202510384s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615514677 (7h4m51.202570918s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615688160 (7h4m51.202744381s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615932236 (7h4m51.202988457s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616184166 (7h4m51.203240378s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616521716 (7h4m51.203577937s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616643614 (7h4m51.203699855s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621231117 (7h4m51.208287338s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621561163 (7h4m51.208617374s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621861323 (7h4m51.208917534s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.622156223 (7h4m51.209212444s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.726040884 (7h4m51.313097094s) [upload=7262]: Next returned nil segment Aug 7 23:31:37.726066772 (7h4m51.313122983s) [upload=7262]: Waiting for error group managing segments... Aug 7 23:31:37.736232517 (7h4m51.323288738s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.737656014 (7h4m51.324712245s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.739754932 (7h4m51.326811153s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.740628113 (7h4m51.327684344s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741712657 (7h4m51.328768867s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741742653 (7h4m51.328798883s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.748242274 (7h4m51.335298505s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.750434577 (7h4m51.337490798s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.758962364 (7h4m51.346018586s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.769693706 (7h4m51.356749917s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.800540734 (7h4m51.387596975s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.924012655 (7h4m51.511068866s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.059695219 (7h4m51.64675144s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102515917 (7h4m51.689572138s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102541606 (7h4m51.689597827s) [seg_pos={0 0}, upload=7262]: Segment reached optimal threshold of 14 pieces. Aug 7 23:31:38.102658574 (7h4m51.689714795s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF Aug 7 23:31:38.102748111 (7h4m51.689804332s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE): protocol: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection Aug 7 23:31:38.102832158 (7h4m51.689888399s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GcyWwKkfavsCsmtq55U72n6XeZKqQiGGYHY9D6UwST8sC14Ac): protocol: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection Aug 7 23:31:38.102967200 (7h4m51.690023421s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12syasUPsyYf4UstQ6cFAr6qamze3vCyAm9dnobk5e83w4bTz69): protocol: context canceled; context canceled Aug 7 23:31:38.102995132 (7h4m51.690051374s) [seg_pos={0 0}, upload=7262]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:31:38.103097553 (7h4m51.690153775s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:31:38.103102412 (7h4m51.690158633s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:31:38.203633212 (7h4m51.790689433s) [upload=7262]: Done uploading segments... Aug 7 23:31:38.203649102 (7h4m51.790705323s) [upload=7262]: Upload finished. err: <nil> Duration: 1.041153565s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 11 out of 60 is failed (threshold=3) [....XXXX.X..X..X......................................XXX.X.] duration: 321.08864ms FAIL (2025-08-07 23:54:40 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 321.108097ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 11 out of 60 is failed (threshold=3) [.....XXXX.X..X..X......................................XXX.X] duration: 30.000572123s FAIL (2025-08-07 23:54:39 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000589657s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.XX.X....X.X..........................................X..] duration: 454.532543ms FAIL (2025-08-07 23:54:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 454.553602ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 12 out of 60 is failed (threshold=3) [...XXXXX.XX............................................XXXXX] duration: 30.001415622s FAIL (2025-08-07 23:54:31 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ctCommit Aug 7 23:46:08.965075092 (7h19m22.552131313s) [upload=7514]: Done uploading segments... Aug 7 23:46:08.965095970 (7h19m22.552152191s) [upload=7514]: Upload finished. err: <nil> Aug 7 23:47:00.017908282 (7h20m13.604964493s) [upload=7527]: Starting upload Aug 7 23:47:00.017955210 (7h20m13.605011411s) [upload=7527]: Done starting upload Aug 7 23:47:00.017998241 (7h20m13.605054452s) [upload=7527]: Uploading segments... Aug 7 23:47:00.018026292 (7h20m13.605082503s) [upload=7527]: Got next segment. Inline: true Aug 7 23:47:00.018029438 (7h20m13.60508565s) [upload=7527]: Waiting for error group managing segments... Aug 7 23:47:00.018058453 (7h20m13.605114664s) [upload=7527]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:47:00.018062390 (7h20m13.605118601s) [upload=7527]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:47:00.018064304 (7h20m13.605120515s) [upload=7527]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:00.075526210 (7h20m13.662582431s) [upload=7527]: Done uploading segments... Aug 7 23:47:00.075540917 (7h20m13.662597138s) [upload=7527]: Upload finished. err: <nil> Aug 7 23:48:09.882765252 (7h21m23.469821463s) [upload=7545]: Starting upload Aug 7 23:48:09.882860198 (7h21m23.46991642s) [upload=7545]: Done starting upload Aug 7 23:48:09.882935729 (7h21m23.469991941s) [upload=7545]: Uploading segments... Aug 7 23:48:09.882965635 (7h21m23.470021856s) [upload=7545]: Got next segment. Inline: true Aug 7 23:48:09.882969723 (7h21m23.470025944s) [upload=7545]: Waiting for error group managing segments... Aug 7 23:48:09.883002865 (7h21m23.470059086s) [upload=7545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:09.883006481 (7h21m23.470062702s) [upload=7545]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:09.883009045 (7h21m23.470065266s) [upload=7545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:09.951272229 (7h21m23.538328451s) [upload=7545]: Done uploading segments... Aug 7 23:48:09.951286926 (7h21m23.538343148s) [upload=7545]: Upload finished. err: <nil> Aug 7 23:48:58.772502770 (7h22m12.359559011s) [upload=7557]: Starting upload Aug 7 23:48:58.772574805 (7h22m12.359631046s) [upload=7557]: Done starting upload Aug 7 23:48:58.772645527 (7h22m12.359701767s) [upload=7557]: Uploading segments... Aug 7 23:48:58.772695300 (7h22m12.359751541s) [upload=7557]: Got next segment. Inline: true Aug 7 23:48:58.772701031 (7h22m12.359757271s) [upload=7557]: Waiting for error group managing segments... Aug 7 23:48:58.772770249 (7h22m12.359826501s) [upload=7557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:58.772780228 (7h22m12.359836469s) [upload=7557]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:58.772787010 (7h22m12.359843252s) [upload=7557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:58.840436652 (7h22m12.427492893s) [upload=7557]: Done uploading segments... Aug 7 23:48:58.840462661 (7h22m12.427518891s) [upload=7557]: Upload finished. err: <nil> Aug 7 23:50:01.971851413 (7h23m15.558907624s) [upload=7574]: Starting upload Aug 7 23:50:01.971890696 (7h23m15.558946907s) [upload=7574]: Done starting upload Aug 7 23:50:01.971977749 (7h23m15.559033961s) [upload=7574]: Uploading segments... Aug 7 23:50:01.972022943 (7h23m15.559079154s) [upload=7574]: Got next segment. Inline: true Aug 7 23:50:01.972026850 (7h23m15.559083062s) [upload=7574]: Waiting for error group managing segments... Aug 7 23:50:01.972060975 (7h23m15.559117185s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:01.972064511 (7h23m15.559120712s) [upload=7574]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:01.972066685 (7h23m15.559122896s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:02.063709030 (7h23m15.650765241s) [upload=7574]: Done uploading segments... Aug 7 23:50:02.063723547 (7h23m15.650779768s) [upload=7574]: Upload finished. err: <nil> Aug 7 23:50:41.160166556 (7h23m54.747222777s) [upload=7588]: Starting upload Aug 7 23:50:41.160220555 (7h23m54.747276777s) [upload=7588]: Done starting upload Aug 7 23:50:41.160264659 (7h23m54.74732088s) [upload=7588]: Uploading segments... Aug 7 23:50:41.160295956 (7h23m54.747352177s) [upload=7588]: Got next segment. Inline: true Aug 7 23:50:41.160299753 (7h23m54.747355964s) [upload=7588]: Waiting for error group managing segments... Aug 7 23:50:41.160331633 (7h23m54.747387844s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:41.160335861 (7h23m54.747392072s) [upload=7588]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:41.160338396 (7h23m54.747394617s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:41.222859735 (7h23m54.809915976s) [upload=7588]: Done uploading segments... Aug 7 23:50:41.222879913 (7h23m54.809936154s) [upload=7588]: Upload finished. err: <nil> Aug 7 23:51:17.131690986 (7h24m30.718747197s) [upload=7599]: Starting upload Aug 7 23:51:17.131738926 (7h24m30.718795147s) [upload=7599]: Done starting upload Aug 7 23:51:17.131792947 (7h24m30.718849178s) [upload=7599]: Uploading segments... Aug 7 23:51:17.131835896 (7h24m30.718892117s) [upload=7599]: Got next segment. Inline: true Aug 7 23:51:17.131842027 (7h24m30.718898239s) [upload=7599]: Waiting for error group managing segments... Aug 7 23:51:17.131883385 (7h24m30.718939627s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:17.131889146 (7h24m30.718945388s) [upload=7599]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:17.132006205 (7h24m30.719062435s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:38.787433326 (7h24m52.374489567s) [upload=7599]: Done uploading segments... Aug 7 23:51:38.787452212 (7h24m52.374508453s) [upload=7599]: Upload finished. err: <nil> Aug 7 23:52:00.256204647 (7h25m13.843260858s) [upload=7610]: Starting upload Aug 7 23:52:00.256263547 (7h25m13.843319758s) [upload=7610]: Done starting upload Aug 7 23:52:00.256316195 (7h25m13.843372406s) [upload=7610]: Uploading segments... Aug 7 23:52:00.256345671 (7h25m13.843401881s) [upload=7610]: Got next segment. Inline: true Aug 7 23:52:00.256349758 (7h25m13.843405969s) [upload=7610]: Waiting for error group managing segments... Aug 7 23:52:00.256399992 (7h25m13.843456203s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:00.256404780 (7h25m13.843460991s) [upload=7610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:00.256407806 (7h25m13.843464017s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:30.237600164 (7h25m43.824656416s) [upload=7610]: Done uploading segments... Aug 7 23:52:30.237630862 (7h25m43.824687113s) [upload=7610]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:53.146251246 (7h26m6.733307456s) [upload=7626]: Starting upload Aug 7 23:52:53.146307339 (7h26m6.733363561s) [upload=7626]: Done starting upload Aug 7 23:52:53.146350962 (7h26m6.733407173s) [upload=7626]: Uploading segments... Aug 7 23:52:53.146411344 (7h26m6.733467565s) [upload=7626]: Got next segment. Inline: true Aug 7 23:52:53.146415271 (7h26m6.733471493s) [upload=7626]: Waiting for error group managing segments... Aug 7 23:52:53.146449966 (7h26m6.733506177s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:53.146454174 (7h26m6.733510395s) [upload=7626]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:53.146456709 (7h26m6.73351293s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.132536521 (7h26m36.719592742s) [upload=7626]: Done uploading segments... Aug 7 23:53:23.132560907 (7h26m36.719617128s) [upload=7626]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:53:28.217132518 (7h26m41.804188759s) [upload=7638]: Starting upload Aug 7 23:53:28.217216185 (7h26m41.804272426s) [upload=7638]: Done starting upload Aug 7 23:53:28.217288009 (7h26m41.80434426s) [upload=7638]: Uploading segments... Aug 7 23:53:28.217336129 (7h26m41.80439237s) [upload=7638]: Got next segment. Inline: true Aug 7 23:53:28.217341890 (7h26m41.804398131s) [upload=7638]: Waiting for error group managing segments... Aug 7 23:53:28.217421438 (7h26m41.804477679s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:28.217427650 (7h26m41.804483901s) [upload=7638]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:28.217431617 (7h26m41.804487859s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:58.204830989 (7h27m11.79188721s) [upload=7638]: Done uploading segments... Aug 7 23:53:58.204849294 (7h27m11.791905505s) [upload=7638]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:01.364416813 (7h27m14.951473054s) [upload=7648]: Starting upload Aug 7 23:54:01.364544481 (7h27m14.951600732s) [upload=7648]: Done starting upload Aug 7 23:54:01.364639518 (7h27m14.95169577s) [upload=7648]: Uploading segments... Aug 7 23:54:01.364703858 (7h27m14.951760109s) [upload=7648]: Got next segment. Inline: true Aug 7 23:54:01.364713517 (7h27m14.951769757s) [upload=7648]: Waiting for error group managing segments... Aug 7 23:54:01.364781374 (7h27m14.951837625s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:01.364791231 (7h27m14.951847472s) [upload=7648]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:01.364797502 (7h27m14.951853754s) [upload=7648]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:31.351186626 (7h27m44.938242847s) [upload=7648]: Done uploading segments... Aug 7 23:54:31.351241118 (7h27m44.938297339s) [upload=7648]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001441521s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........XXX.X.X.........................................XX.] duration: 1.423643444s FAIL (2025-08-07 23:54:21 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ading piece... Aug 7 23:51:17.347399839 (7h24m30.934456051s) [noise=true , address=104.36.82.35:4611 , piece_id=FE47XNPAFZAELTDVIO6C4URDMXRWEXCM37BFM4LANQQ6QDSU7HAQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347417763 (7h24m30.934473974s) [noise=true , address=104.36.82.4:4806 , piece_id=X42UHVJVQMP46Y7QDE377JH6FNETULCU4JBDSGTVJDRSUE5LAVOA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347625520 (7h24m30.934681731s) [noise=true , address=45.91.116.10:4098 , piece_id=5ITNSAGAAD2QAEAZGQSRYU2GSIS54FPDHU6HETL6QVR57KUDYICQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347755754 (7h24m30.934811985s) [noise=true , address=104.36.82.45:4921 , piece_id=GQXCWGC6EHWZONLT426WVNPLASR7SECUANPQGGZNMASCRRD4X4SQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347968761 (7h24m30.935024982s) [noise=true , address=104.36.82.62:4438 , piece_id=3ND3RVOIIEJMN5GH7KZRHPYVNIH5KB62OU6KEV5DVOLCAZI4BQLA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348133560 (7h24m30.935189781s) [noise=true , address=104.36.82.39:4729 , piece_id=B7IGTQUWZAQNQDUYOUAB3WGYK5QHKXXJKRDGKMEGDK6KS4ZFC5GA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348193100 (7h24m30.935249311s) [noise=true , address=104.36.82.23:4475 , piece_id=LZMIGPR677ZHFYTXZJAMLW3GC3XLON7LGPRRBWZWVEXG4GGLEEEA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348536501 (7h24m30.935592722s) [noise=true , address=104.36.82.61:4406 , piece_id=N63ZWSHAILE55ZOUL5H6RYDUTB5F2F3EWMDT52RPJ5F46MLKRDMA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348656385 (7h24m30.935712596s) [noise=true , address=104.36.82.12:4178 , piece_id=NQWZGYOLKX6SHV3RXIXVGFDAXFP4VMFMP5FJKQPZ63YZ2NK3RJDQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348854374 (7h24m30.935910585s) [noise=true , address=104.36.82.15:4288 , piece_id=KWW7E4ZMO42C3YEAH3WBCEYSW6TTNDX7TQ4MEITLSC2RR2CE4VDA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348919236 (7h24m30.935975457s) [noise=true , address=104.36.82.36:4649 , piece_id=NV2V22M4HHIEHH4RENXMFHFYXHQF7LPJRPPD7ZBKHPSZFKI2BVTA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.430583697 (7h24m31.017639918s) [upload=7600]: Next returned nil segment Aug 7 23:51:17.430596350 (7h24m31.017652571s) [upload=7600]: Waiting for error group managing segments... Aug 7 23:51:17.450496095 (7h24m31.037552306s) [noise=true , address=104.36.82.39:4729 , piece_id=B7IGTQUWZAQNQDUYOUAB3WGYK5QHKXXJKRDGKMEGDK6KS4ZFC5GA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.457273115 (7h24m31.044329336s) [noise=true , address=45.91.116.11:4121 , piece_id=UI4HOUG4KLWC3LFGDVZ3C3IUMB2DGJP2QOTIOLWS5C2HUAMSATMA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.458658320 (7h24m31.045714532s) [noise=true , address=104.36.82.36:4649 , piece_id=NV2V22M4HHIEHH4RENXMFHFYXHQF7LPJRPPD7ZBKHPSZFKI2BVTA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.464324767 (7h24m31.051380978s) [noise=true , address=104.36.82.62:4438 , piece_id=3ND3RVOIIEJMN5GH7KZRHPYVNIH5KB62OU6KEV5DVOLCAZI4BQLA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.469258046 (7h24m31.056314256s) [noise=true , address=104.36.82.4:4806 , piece_id=X42UHVJVQMP46Y7QDE377JH6FNETULCU4JBDSGTVJDRSUE5LAVOA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.470547041 (7h24m31.057603252s) [noise=true , address=45.91.116.10:4098 , piece_id=5ITNSAGAAD2QAEAZGQSRYU2GSIS54FPDHU6HETL6QVR57KUDYICQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.482895480 (7h24m31.069951691s) [noise=true , address=104.36.82.59:4327 , piece_id=5AM7TVDV5ORD5WPHKEHOM652JETCJ6VH3JOJ6HPJTIOTABTYVELA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.503848269 (7h24m31.090904481s) [noise=true , address=104.36.82.61:4406 , piece_id=N63ZWSHAILE55ZOUL5H6RYDUTB5F2F3EWMDT52RPJ5F46MLKRDMA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.589438619 (7h24m31.17649484s) [noise=true , address=79.127.201.215:4343 , piece_id=6IVWBNSJNI4ZAEIWFTRYUHIHZ35YTTZF3FTEF7G76YEBUP2NBJFA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.633863581 (7h24m31.220919802s) [noise=true , address=104.36.82.42:4801 , piece_id=ILNDV4CZCPEST6R4DUTSHQ5OJS2N4BLPRLDYIYNBTZKRR4RSCOFA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.636493760 (7h24m31.22354997s) [noise=true , address=104.36.82.44:4904 , piece_id=TFXFZ53DGPOVVXTTZMRMTJR53DGXNHWUF43I7OCC4YNCRW4QUCKA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.637613389 (7h24m31.224669611s) [noise=true , address=104.36.82.23:4475 , piece_id=LZMIGPR677ZHFYTXZJAMLW3GC3XLON7LGPRRBWZWVEXG4GGLEEEA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.654845844 (7h24m31.241902065s) [noise=true , address=104.36.82.22:4524 , piece_id=66YIUJKSDV52T5NMMFHYCZH7L34UKH3RLIS5WZ3VWRBL3KEE7UWQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.853873712 (7h24m31.440929933s) [noise=true , address=104.36.82.12:4178 , piece_id=NQWZGYOLKX6SHV3RXIXVGFDAXFP4VMFMP5FJKQPZ63YZ2NK3RJDQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.853888850 (7h24m31.440945061s) [seg_pos={0 0}, upload=7600]: Segment reached optimal threshold of 14 pieces. Aug 7 23:51:17.853987013 (7h24m31.441043234s) [noise=true , address=104.36.82.45:4921 , piece_id=GQXCWGC6EHWZONLT426WVNPLASR7SECUANPQGGZNMASCRRD4X4SQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bBRKXiuSN2cw6KrdP8q4Dsf4MiFCUSgMEGjqG2dKL9RLTQra1): protocol: expected piece hash; context canceled; EOF Aug 7 23:51:17.854295209 (7h24m31.44135142s) [noise=true , address=104.36.82.46:4499 , piece_id=6OBTEQJKT4AJRGUT27EYWNLG4E2TKBCQBRGXTTEOU4SJY2SWUTFQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc): protocol: write tcp 5.161.89.252:63506->104.36.82.46:4499: use of closed network connection; write tcp 5.161.89.252:63506->104.36.82.46:4499: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63506->104.36.82.46:4499: use of closed network connection Aug 7 23:51:17.854338279 (7h24m31.44139449s) [noise=true , address=104.36.82.35:4611 , piece_id=FE47XNPAFZAELTDVIO6C4URDMXRWEXCM37BFM4LANQQ6QDSU7HAQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zkqEMSKx7zSHPFNZ51b5TLzL2cFQL8VyxAG14Lg1pT8zAu3tc): protocol: write tcp 5.161.89.252:46266->104.36.82.35:4611: use of closed network connection; write tcp 5.161.89.252:46266->104.36.82.35:4611: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:46266->104.36.82.35:4611: use of closed network connection Aug 7 23:51:17.854399554 (7h24m31.441455765s) [noise=true , address=104.36.82.15:4288 , piece_id=KWW7E4ZMO42C3YEAH3WBCEYSW6TTNDX7TQ4MEITLSC2RR2CE4VDA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gPjPGuueZnizLT2QvJN9y7riwKiW5TWMkQtmGGPP7VZvkNoB7): protocol: read tcp 5.161.89.252:35428->104.36.82.15:4288: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:51:17.854419941 (7h24m31.441476162s) [seg_pos={0 0}, upload=7600]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:51:17.854472530 (7h24m31.441528751s) [upload=7600]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:51:17.854476738 (7h24m31.441532959s) [upload=7600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:17.900817215 (7h24m31.487873465s) [upload=7600]: Done uploading segments... Aug 7 23:51:17.900834827 (7h24m31.487891079s) [upload=7600]: Upload finished. err: <nil> Aug 7 23:52:11.382772324 (7h25m24.969828534s) [upload=7616]: Starting upload Aug 7 23:52:11.382828459 (7h25m24.96988468s) [upload=7616]: Done starting upload Aug 7 23:52:11.382871529 (7h25m24.96992774s) [upload=7616]: Uploading segments... Aug 7 23:52:11.383053639 (7h25m24.97010986s) [upload=7616]: Got next segment. Inline: false Aug 7 23:52:11.383080680 (7h25m24.970136891s) [upload=7616]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:11.383084406 (7h25m24.970140617s) [upload=7616]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:11.371970391 (7h26m24.959026631s) [upload=7616]: Next segment err: context deadline exceeded Aug 7 23:53:11.372056972 (7h26m24.959113223s) [upload=7616]: Done uploading segments... Aug 7 23:53:11.372090856 (7h26m24.959147096s) [upload=7616]: Upload finished. err: context deadline exceeded Aug 7 23:53:11.462534524 (7h26m25.049590775s) [upload=7633]: Starting upload Aug 7 23:53:11.462662182 (7h26m25.049718423s) [upload=7633]: Done starting upload Aug 7 23:53:11.462730148 (7h26m25.0497864s) [upload=7633]: Uploading segments... Aug 7 23:53:11.463218160 (7h26m25.050274451s) [upload=7633]: Got next segment. Inline: false Aug 7 23:53:11.463267292 (7h26m25.050323533s) [upload=7633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:11.463276689 (7h26m25.050332931s) [upload=7633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:54:11.448954917 (7h27m25.036011128s) [upload=7633]: Next segment err: context deadline exceeded Aug 7 23:54:11.449030117 (7h27m25.036086328s) [upload=7633]: Done uploading segments... Aug 7 23:54:11.449047229 (7h27m25.03610344s) [upload=7633]: Upload finished. err: context deadline exceeded Duration: 1.423672139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...XXXXXX.X.X...........................................X.X.] duration: 2.011533337s FAIL (2025-08-07 23:54:16 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.011552763s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 153.8332ms FAIL (2025-08-07 23:54:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 153.883815ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [......X.XXXXX..X............................................] duration: 199.462919ms FAIL (2025-08-07 23:54:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 199.48029ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 874.830401ms FAIL (2025-08-07 23:54:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== 1345s) [upload=7262]: Got next segment. Inline: false Aug 7 23:31:37.583906637 (7h4m51.170962878s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:31:37.583953414 (7h4m51.171009656s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:31:37.613963041 (7h4m51.201019262s) [seg_pos={0 0}, upload=7262]: Begin upload segment... Aug 7 23:31:37.614008966 (7h4m51.201065176s) [seg_pos={0 0}, upload=7262]: Done begin upload segment. Aug 7 23:31:37.614029103 (7h4m51.201085324s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614108311 (7h4m51.201164542s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614344612 (7h4m51.201400833s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614634754 (7h4m51.201690975s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614919926 (7h4m51.201976137s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.614943150 (7h4m51.201999391s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615146719 (7h4m51.20220294s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615454163 (7h4m51.202510384s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615514677 (7h4m51.202570918s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615688160 (7h4m51.202744381s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.615932236 (7h4m51.202988457s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616184166 (7h4m51.203240378s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616521716 (7h4m51.203577937s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.616643614 (7h4m51.203699855s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621231117 (7h4m51.208287338s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621561163 (7h4m51.208617374s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.621861323 (7h4m51.208917534s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.622156223 (7h4m51.209212444s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Uploading piece... Aug 7 23:31:37.726040884 (7h4m51.313097094s) [upload=7262]: Next returned nil segment Aug 7 23:31:37.726066772 (7h4m51.313122983s) [upload=7262]: Waiting for error group managing segments... Aug 7 23:31:37.736232517 (7h4m51.323288738s) [noise=true , address=45.91.116.4:4959 , piece_id=GCAOGJC2EERNOH7Q4NLCGMX5BJHPPPWAKLJVSS3LK6Z52O6YCYCA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.737656014 (7h4m51.324712245s) [noise=true , address=104.36.82.43:4845 , piece_id=7J5PS3HMVPBRBNGHURPFCUVZIL2XD7IDAMDFSA3PJZ56FD2WV3IA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.739754932 (7h4m51.326811153s) [noise=true , address=104.36.82.22:4515 , piece_id=ULUTYRTCHVD6CVJLXHJ5GQATQPLPRXOVHPRDQJ6MZU2R2P66NDEQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.740628113 (7h4m51.327684344s) [noise=true , address=104.36.82.47:4536 , piece_id=ROPDHSIVOPNHGPWV7DZCZ3GZGJQQB2Q76VBSH53LDXDBEXPTN3NA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741712657 (7h4m51.328768867s) [noise=true , address=104.36.82.45:4939 , piece_id=2HHS67OS52MHE4VGVXJOM6NXWIPYSPHSH5GJN44WU5S7B4S22QNQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.741742653 (7h4m51.328798883s) [noise=true , address=104.36.82.37:4680 , piece_id=P7NUR3BGWKRH5SPUDCLHY6KYZR3NYQ5RJ4NIUJVCRCDRX7CP26FQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.748242274 (7h4m51.335298505s) [noise=true , address=104.36.82.14:4231 , piece_id=AYWBPI67PZRBJ4ZFXKJS5FA7QJQZBE33SSOIK3KEO3SYKCZXNLKA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.750434577 (7h4m51.337490798s) [noise=true , address=104.36.82.23:4468 , piece_id=3SMMMVEKAVPJEYKM3FL5KIVHFEUANKX47MUBOGEVZJPOFU424WGQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.758962364 (7h4m51.346018586s) [noise=true , address=45.91.116.5:4306 , piece_id=3GL3EHAQ4K7LFWOSNFOCD6RVKR5IXLKGLIYKDSUWHQ2ADPGC2OHA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.769693706 (7h4m51.356749917s) [noise=true , address=104.36.82.54:4179 , piece_id=I4456FEQNQF43VTACXRT2EVEFKDAC4ID3V4D562ESLSO3H625KKQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.800540734 (7h4m51.387596975s) [noise=true , address=104.36.82.61:4378 , piece_id=RK2UJMQB7UJR7V5C656ZNU43QWYKK6D7VATSO6ZMOL5JMUG63LAQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:37.924012655 (7h4m51.511068866s) [noise=true , address=104.36.82.59:4327 , piece_id=B4CO5NTMWI3RASH6VSJKGZI6GPJIBNLPL5XM2SS7BQG7WEOQ2WWQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.059695219 (7h4m51.64675144s) [noise=true , address=79.127.201.215:4348 , piece_id=MC5JXBACOI5RB23WMS4ZZM3BQ5OKOBHKQ5MRWT7FHMT7BPE7MT6A, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102515917 (7h4m51.689572138s) [noise=true , address=79.127.205.234:4600 , piece_id=EEIYWE3SXSTJXAM3HSYWJYQJRZCEJTGUF3RSOSIFJLY2BWZS3MMA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: <nil> Aug 7 23:31:38.102541606 (7h4m51.689597827s) [seg_pos={0 0}, upload=7262]: Segment reached optimal threshold of 14 pieces. Aug 7 23:31:38.102658574 (7h4m51.689714795s) [noise=true , address=64.49.13.27:4964 , piece_id=5AN2UI36U4PRJRZAKA7HJ7V62B5PVDRFKRBH65G7RP5YUEZEF3XA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF Aug 7 23:31:38.102748111 (7h4m51.689804332s) [noise=true , address=64.49.13.26:4953 , piece_id=HBQDTPFQQCVMKO5F2NQECZU52ZLFWWA52LIL7S7D7B7VGRQR5NAA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE): protocol: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39408->64.49.13.26:4953: use of closed network connection Aug 7 23:31:38.102832158 (7h4m51.689888399s) [noise=true , address=104.36.82.16:4315 , piece_id=TV2LFU35KXULNBE3QMOVAX752IFJTJGQNE633YN5RZVJXOUCNJTQ, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GcyWwKkfavsCsmtq55U72n6XeZKqQiGGYHY9D6UwST8sC14Ac): protocol: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:18248->104.36.82.16:4315: use of closed network connection Aug 7 23:31:38.102967200 (7h4m51.690023421s) [noise=true , address=104.36.82.62:4416 , piece_id=LPWXXMNM4JO55TVC7UFVFC6ZRI3FZWEJR3P6LL7ZVDW7PJJUJUWA, seg_pos={0 0}, upload=7262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12syasUPsyYf4UstQ6cFAr6qamze3vCyAm9dnobk5e83w4bTz69): protocol: context canceled; context canceled Aug 7 23:31:38.102995132 (7h4m51.690051374s) [seg_pos={0 0}, upload=7262]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:31:38.103097553 (7h4m51.690153775s) [upload=7262]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:31:38.103102412 (7h4m51.690158633s) [upload=7262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:31:38.203633212 (7h4m51.790689433s) [upload=7262]: Done uploading segments... Aug 7 23:31:38.203649102 (7h4m51.790705323s) [upload=7262]: Upload finished. err: <nil> Duration: 874.854346ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........XXX.X.X.........................................XX] duration: 1m0.00046619s FAIL (2025-08-07 23:54:11 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ading piece... Aug 7 23:51:17.347399839 (7h24m30.934456051s) [noise=true , address=104.36.82.35:4611 , piece_id=FE47XNPAFZAELTDVIO6C4URDMXRWEXCM37BFM4LANQQ6QDSU7HAQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347417763 (7h24m30.934473974s) [noise=true , address=104.36.82.4:4806 , piece_id=X42UHVJVQMP46Y7QDE377JH6FNETULCU4JBDSGTVJDRSUE5LAVOA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347625520 (7h24m30.934681731s) [noise=true , address=45.91.116.10:4098 , piece_id=5ITNSAGAAD2QAEAZGQSRYU2GSIS54FPDHU6HETL6QVR57KUDYICQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347755754 (7h24m30.934811985s) [noise=true , address=104.36.82.45:4921 , piece_id=GQXCWGC6EHWZONLT426WVNPLASR7SECUANPQGGZNMASCRRD4X4SQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347968761 (7h24m30.935024982s) [noise=true , address=104.36.82.62:4438 , piece_id=3ND3RVOIIEJMN5GH7KZRHPYVNIH5KB62OU6KEV5DVOLCAZI4BQLA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348133560 (7h24m30.935189781s) [noise=true , address=104.36.82.39:4729 , piece_id=B7IGTQUWZAQNQDUYOUAB3WGYK5QHKXXJKRDGKMEGDK6KS4ZFC5GA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348193100 (7h24m30.935249311s) [noise=true , address=104.36.82.23:4475 , piece_id=LZMIGPR677ZHFYTXZJAMLW3GC3XLON7LGPRRBWZWVEXG4GGLEEEA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348536501 (7h24m30.935592722s) [noise=true , address=104.36.82.61:4406 , piece_id=N63ZWSHAILE55ZOUL5H6RYDUTB5F2F3EWMDT52RPJ5F46MLKRDMA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348656385 (7h24m30.935712596s) [noise=true , address=104.36.82.12:4178 , piece_id=NQWZGYOLKX6SHV3RXIXVGFDAXFP4VMFMP5FJKQPZ63YZ2NK3RJDQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348854374 (7h24m30.935910585s) [noise=true , address=104.36.82.15:4288 , piece_id=KWW7E4ZMO42C3YEAH3WBCEYSW6TTNDX7TQ4MEITLSC2RR2CE4VDA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348919236 (7h24m30.935975457s) [noise=true , address=104.36.82.36:4649 , piece_id=NV2V22M4HHIEHH4RENXMFHFYXHQF7LPJRPPD7ZBKHPSZFKI2BVTA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.430583697 (7h24m31.017639918s) [upload=7600]: Next returned nil segment Aug 7 23:51:17.430596350 (7h24m31.017652571s) [upload=7600]: Waiting for error group managing segments... Aug 7 23:51:17.450496095 (7h24m31.037552306s) [noise=true , address=104.36.82.39:4729 , piece_id=B7IGTQUWZAQNQDUYOUAB3WGYK5QHKXXJKRDGKMEGDK6KS4ZFC5GA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.457273115 (7h24m31.044329336s) [noise=true , address=45.91.116.11:4121 , piece_id=UI4HOUG4KLWC3LFGDVZ3C3IUMB2DGJP2QOTIOLWS5C2HUAMSATMA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.458658320 (7h24m31.045714532s) [noise=true , address=104.36.82.36:4649 , piece_id=NV2V22M4HHIEHH4RENXMFHFYXHQF7LPJRPPD7ZBKHPSZFKI2BVTA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.464324767 (7h24m31.051380978s) [noise=true , address=104.36.82.62:4438 , piece_id=3ND3RVOIIEJMN5GH7KZRHPYVNIH5KB62OU6KEV5DVOLCAZI4BQLA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.469258046 (7h24m31.056314256s) [noise=true , address=104.36.82.4:4806 , piece_id=X42UHVJVQMP46Y7QDE377JH6FNETULCU4JBDSGTVJDRSUE5LAVOA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.470547041 (7h24m31.057603252s) [noise=true , address=45.91.116.10:4098 , piece_id=5ITNSAGAAD2QAEAZGQSRYU2GSIS54FPDHU6HETL6QVR57KUDYICQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.482895480 (7h24m31.069951691s) [noise=true , address=104.36.82.59:4327 , piece_id=5AM7TVDV5ORD5WPHKEHOM652JETCJ6VH3JOJ6HPJTIOTABTYVELA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.503848269 (7h24m31.090904481s) [noise=true , address=104.36.82.61:4406 , piece_id=N63ZWSHAILE55ZOUL5H6RYDUTB5F2F3EWMDT52RPJ5F46MLKRDMA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.589438619 (7h24m31.17649484s) [noise=true , address=79.127.201.215:4343 , piece_id=6IVWBNSJNI4ZAEIWFTRYUHIHZ35YTTZF3FTEF7G76YEBUP2NBJFA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.633863581 (7h24m31.220919802s) [noise=true , address=104.36.82.42:4801 , piece_id=ILNDV4CZCPEST6R4DUTSHQ5OJS2N4BLPRLDYIYNBTZKRR4RSCOFA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.636493760 (7h24m31.22354997s) [noise=true , address=104.36.82.44:4904 , piece_id=TFXFZ53DGPOVVXTTZMRMTJR53DGXNHWUF43I7OCC4YNCRW4QUCKA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.637613389 (7h24m31.224669611s) [noise=true , address=104.36.82.23:4475 , piece_id=LZMIGPR677ZHFYTXZJAMLW3GC3XLON7LGPRRBWZWVEXG4GGLEEEA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.654845844 (7h24m31.241902065s) [noise=true , address=104.36.82.22:4524 , piece_id=66YIUJKSDV52T5NMMFHYCZH7L34UKH3RLIS5WZ3VWRBL3KEE7UWQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.853873712 (7h24m31.440929933s) [noise=true , address=104.36.82.12:4178 , piece_id=NQWZGYOLKX6SHV3RXIXVGFDAXFP4VMFMP5FJKQPZ63YZ2NK3RJDQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.853888850 (7h24m31.440945061s) [seg_pos={0 0}, upload=7600]: Segment reached optimal threshold of 14 pieces. Aug 7 23:51:17.853987013 (7h24m31.441043234s) [noise=true , address=104.36.82.45:4921 , piece_id=GQXCWGC6EHWZONLT426WVNPLASR7SECUANPQGGZNMASCRRD4X4SQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bBRKXiuSN2cw6KrdP8q4Dsf4MiFCUSgMEGjqG2dKL9RLTQra1): protocol: expected piece hash; context canceled; EOF Aug 7 23:51:17.854295209 (7h24m31.44135142s) [noise=true , address=104.36.82.46:4499 , piece_id=6OBTEQJKT4AJRGUT27EYWNLG4E2TKBCQBRGXTTEOU4SJY2SWUTFQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc): protocol: write tcp 5.161.89.252:63506->104.36.82.46:4499: use of closed network connection; write tcp 5.161.89.252:63506->104.36.82.46:4499: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63506->104.36.82.46:4499: use of closed network connection Aug 7 23:51:17.854338279 (7h24m31.44139449s) [noise=true , address=104.36.82.35:4611 , piece_id=FE47XNPAFZAELTDVIO6C4URDMXRWEXCM37BFM4LANQQ6QDSU7HAQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zkqEMSKx7zSHPFNZ51b5TLzL2cFQL8VyxAG14Lg1pT8zAu3tc): protocol: write tcp 5.161.89.252:46266->104.36.82.35:4611: use of closed network connection; write tcp 5.161.89.252:46266->104.36.82.35:4611: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:46266->104.36.82.35:4611: use of closed network connection Aug 7 23:51:17.854399554 (7h24m31.441455765s) [noise=true , address=104.36.82.15:4288 , piece_id=KWW7E4ZMO42C3YEAH3WBCEYSW6TTNDX7TQ4MEITLSC2RR2CE4VDA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gPjPGuueZnizLT2QvJN9y7riwKiW5TWMkQtmGGPP7VZvkNoB7): protocol: read tcp 5.161.89.252:35428->104.36.82.15:4288: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:51:17.854419941 (7h24m31.441476162s) [seg_pos={0 0}, upload=7600]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:51:17.854472530 (7h24m31.441528751s) [upload=7600]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:51:17.854476738 (7h24m31.441532959s) [upload=7600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:17.900817215 (7h24m31.487873465s) [upload=7600]: Done uploading segments... Aug 7 23:51:17.900834827 (7h24m31.487891079s) [upload=7600]: Upload finished. err: <nil> Aug 7 23:52:11.382772324 (7h25m24.969828534s) [upload=7616]: Starting upload Aug 7 23:52:11.382828459 (7h25m24.96988468s) [upload=7616]: Done starting upload Aug 7 23:52:11.382871529 (7h25m24.96992774s) [upload=7616]: Uploading segments... Aug 7 23:52:11.383053639 (7h25m24.97010986s) [upload=7616]: Got next segment. Inline: false Aug 7 23:52:11.383080680 (7h25m24.970136891s) [upload=7616]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:11.383084406 (7h25m24.970140617s) [upload=7616]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:11.371970391 (7h26m24.959026631s) [upload=7616]: Next segment err: context deadline exceeded Aug 7 23:53:11.372056972 (7h26m24.959113223s) [upload=7616]: Done uploading segments... Aug 7 23:53:11.372090856 (7h26m24.959147096s) [upload=7616]: Upload finished. err: context deadline exceeded Aug 7 23:53:11.462534524 (7h26m25.049590775s) [upload=7633]: Starting upload Aug 7 23:53:11.462662182 (7h26m25.049718423s) [upload=7633]: Done starting upload Aug 7 23:53:11.462730148 (7h26m25.0497864s) [upload=7633]: Uploading segments... Aug 7 23:53:11.463218160 (7h26m25.050274451s) [upload=7633]: Got next segment. Inline: false Aug 7 23:53:11.463267292 (7h26m25.050323533s) [upload=7633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:11.463276689 (7h26m25.050332931s) [upload=7633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:54:11.448954917 (7h27m25.036011128s) [upload=7633]: Next segment err: context deadline exceeded Aug 7 23:54:11.449030117 (7h27m25.036086328s) [upload=7633]: Done uploading segments... Aug 7 23:54:11.449047229 (7h27m25.03610344s) [upload=7633]: Upload finished. err: context deadline exceeded Duration: 1m0.000480166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [....XXXXXX.X...............................................X] duration: 1m0.001179591s FAIL (2025-08-07 23:54:11 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :45:31.386895707 (7h18m44.973951958s) [upload=7501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:45:31.448582950 (7h18m45.035639191s) [upload=7501]: Done uploading segments... Aug 7 23:45:31.448602987 (7h18m45.035659229s) [upload=7501]: Upload finished. err: <nil> Aug 7 23:46:05.500265801 (7h19m19.087322041s) [upload=7512]: Starting upload Aug 7 23:46:05.500376477 (7h19m19.087432728s) [upload=7512]: Done starting upload Aug 7 23:46:05.500456777 (7h19m19.087513028s) [upload=7512]: Uploading segments... Aug 7 23:46:05.500516349 (7h19m19.08757266s) [upload=7512]: Got next segment. Inline: true Aug 7 23:46:05.500526007 (7h19m19.087582288s) [upload=7512]: Waiting for error group managing segments... Aug 7 23:46:05.500594884 (7h19m19.087651126s) [upload=7512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:05.500603801 (7h19m19.087660053s) [upload=7512]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:05.500610143 (7h19m19.087666395s) [upload=7512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:05.713026270 (7h19m19.300082522s) [upload=7512]: Done uploading segments... Aug 7 23:46:05.713042230 (7h19m19.300098472s) [upload=7512]: Upload finished. err: <nil> Aug 7 23:46:59.827069649 (7h20m13.41412587s) [upload=7526]: Starting upload Aug 7 23:46:59.827112839 (7h20m13.414169051s) [upload=7526]: Done starting upload Aug 7 23:46:59.827158284 (7h20m13.414214506s) [upload=7526]: Uploading segments... Aug 7 23:46:59.827184925 (7h20m13.414241135s) [upload=7526]: Got next segment. Inline: true Aug 7 23:46:59.827188771 (7h20m13.414244992s) [upload=7526]: Waiting for error group managing segments... Aug 7 23:46:59.827219097 (7h20m13.414275308s) [upload=7526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:59.827222835 (7h20m13.414279056s) [upload=7526]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:59.827225119 (7h20m13.414281329s) [upload=7526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:59.879507639 (7h20m13.46656385s) [upload=7526]: Done uploading segments... Aug 7 23:46:59.879523058 (7h20m13.466579269s) [upload=7526]: Upload finished. err: <nil> Aug 7 23:47:39.924312697 (7h20m53.511368928s) [upload=7536]: Starting upload Aug 7 23:47:39.924413405 (7h20m53.511469646s) [upload=7536]: Done starting upload Aug 7 23:47:39.924491079 (7h20m53.51154731s) [upload=7536]: Uploading segments... Aug 7 23:47:39.924700320 (7h20m53.511756551s) [upload=7536]: Got next segment. Inline: true Aug 7 23:47:39.924711842 (7h20m53.511768063s) [upload=7536]: Waiting for error group managing segments... Aug 7 23:47:39.924828248 (7h20m53.51188448s) [upload=7536]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:47:39.924836915 (7h20m53.511893156s) [upload=7536]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:47:39.924842024 (7h20m53.511898255s) [upload=7536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:39.988158721 (7h20m53.575214932s) [upload=7536]: Done uploading segments... Aug 7 23:47:39.988176785 (7h20m53.575232995s) [upload=7536]: Upload finished. err: <nil> Aug 7 23:48:17.585904864 (7h21m31.172961076s) [upload=7548]: Starting upload Aug 7 23:48:17.585990855 (7h21m31.173047066s) [upload=7548]: Done starting upload Aug 7 23:48:17.586065835 (7h21m31.173122035s) [upload=7548]: Uploading segments... Aug 7 23:48:17.586102964 (7h21m31.173159205s) [upload=7548]: Got next segment. Inline: true Aug 7 23:48:17.586107262 (7h21m31.173163473s) [upload=7548]: Waiting for error group managing segments... Aug 7 23:48:17.586144451 (7h21m31.173200662s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:17.586148969 (7h21m31.17320518s) [upload=7548]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:17.586151854 (7h21m31.173208066s) [upload=7548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:17.642355873 (7h21m31.229412094s) [upload=7548]: Done uploading segments... Aug 7 23:48:17.642371582 (7h21m31.229427794s) [upload=7548]: Upload finished. err: <nil> Aug 7 23:49:19.929340015 (7h22m33.516396255s) [upload=7565]: Starting upload Aug 7 23:49:19.929497669 (7h22m33.51655391s) [upload=7565]: Done starting upload Aug 7 23:49:19.929593468 (7h22m33.516649709s) [upload=7565]: Uploading segments... Aug 7 23:49:19.929640947 (7h22m33.516697188s) [upload=7565]: Got next segment. Inline: true Aug 7 23:49:19.929647679 (7h22m33.516703931s) [upload=7565]: Waiting for error group managing segments... Aug 7 23:49:19.929695078 (7h22m33.516751329s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:19.929701059 (7h22m33.5167573s) [upload=7565]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:19.929704956 (7h22m33.516761197s) [upload=7565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:19.992636681 (7h22m33.579692922s) [upload=7565]: Done uploading segments... Aug 7 23:49:19.992654385 (7h22m33.579710625s) [upload=7565]: Upload finished. err: <nil> Aug 7 23:50:07.176468149 (7h23m20.76352438s) [upload=7576]: Starting upload Aug 7 23:50:07.176537798 (7h23m20.763594019s) [upload=7576]: Done starting upload Aug 7 23:50:07.176592070 (7h23m20.763648311s) [upload=7576]: Uploading segments... Aug 7 23:50:07.176626473 (7h23m20.763682705s) [upload=7576]: Got next segment. Inline: true Aug 7 23:50:07.176631002 (7h23m20.763687213s) [upload=7576]: Waiting for error group managing segments... Aug 7 23:50:07.176664084 (7h23m20.763720295s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:07.176668132 (7h23m20.763724343s) [upload=7576]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:07.176670586 (7h23m20.763726807s) [upload=7576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:07.240851653 (7h23m20.827907864s) [upload=7576]: Done uploading segments... Aug 7 23:50:07.240866771 (7h23m20.827922992s) [upload=7576]: Upload finished. err: <nil> Aug 7 23:50:45.893754262 (7h23m59.480810504s) [upload=7591]: Starting upload Aug 7 23:50:45.893825847 (7h23m59.480882088s) [upload=7591]: Done starting upload Aug 7 23:50:45.893912749 (7h23m59.48096899s) [upload=7591]: Uploading segments... Aug 7 23:50:45.893955949 (7h23m59.4810122s) [upload=7591]: Got next segment. Inline: true Aug 7 23:50:45.893961970 (7h23m59.481018211s) [upload=7591]: Waiting for error group managing segments... Aug 7 23:50:45.894005482 (7h23m59.481061723s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:45.894011062 (7h23m59.481067304s) [upload=7591]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:45.894014729 (7h23m59.48107097s) [upload=7591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:05.079621048 (7h24m18.666677259s) [upload=7591]: Done uploading segments... Aug 7 23:51:05.079636297 (7h24m18.666692508s) [upload=7591]: Upload finished. err: <nil> Aug 7 23:51:21.725622238 (7h24m35.312678469s) [upload=7603]: Starting upload Aug 7 23:51:21.725696627 (7h24m35.312752838s) [upload=7603]: Done starting upload Aug 7 23:51:21.725758231 (7h24m35.312814443s) [upload=7603]: Uploading segments... Aug 7 23:51:21.725796774 (7h24m35.312852985s) [upload=7603]: Got next segment. Inline: true Aug 7 23:51:21.725802845 (7h24m35.312859056s) [upload=7603]: Waiting for error group managing segments... Aug 7 23:51:21.725848871 (7h24m35.312905082s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:21.725855313 (7h24m35.312911523s) [upload=7603]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:21.725859100 (7h24m35.312915351s) [upload=7603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:21.784505948 (7h24m35.371562158s) [upload=7603]: Done uploading segments... Aug 7 23:51:21.784521767 (7h24m35.371577978s) [upload=7603]: Upload finished. err: <nil> Aug 7 23:52:10.285208176 (7h25m23.872264397s) [upload=7615]: Starting upload Aug 7 23:52:10.285263228 (7h25m23.872319449s) [upload=7615]: Done starting upload Aug 7 23:52:10.285310727 (7h25m23.872366937s) [upload=7615]: Uploading segments... Aug 7 23:52:10.285339711 (7h25m23.872395932s) [upload=7615]: Got next segment. Inline: true Aug 7 23:52:10.285343438 (7h25m23.872399649s) [upload=7615]: Waiting for error group managing segments... Aug 7 23:52:10.285391158 (7h25m23.872447379s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:10.285395065 (7h25m23.872451276s) [upload=7615]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:10.285397770 (7h25m23.872453991s) [upload=7615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:10.400888570 (7h25m23.987944781s) [upload=7615]: Done uploading segments... Aug 7 23:52:10.400904530 (7h25m23.987960741s) [upload=7615]: Upload finished. err: <nil> Aug 7 23:53:11.795016020 (7h26m25.382072262s) [upload=7635]: Starting upload Aug 7 23:53:11.795108102 (7h26m25.382164354s) [upload=7635]: Done starting upload Aug 7 23:53:11.795175588 (7h26m25.382231829s) [upload=7635]: Uploading segments... Aug 7 23:53:11.795254645 (7h26m25.382310886s) [upload=7635]: Got next segment. Inline: true Aug 7 23:53:11.795262130 (7h26m25.38231837s) [upload=7635]: Waiting for error group managing segments... Aug 7 23:53:11.795319296 (7h26m25.382375538s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:11.795326170 (7h26m25.38238242s) [upload=7635]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:11.795329857 (7h26m25.382386097s) [upload=7635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:11.782974743 (7h27m25.370030954s) [upload=7635]: Done uploading segments... Aug 7 23:54:11.782991474 (7h27m25.370047695s) [upload=7635]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1m0.001205869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....XXXXXX.X.X...........................................X.X] duration: 30.001101184s FAIL (2025-08-07 23:54:09 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001122374s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.XX.X....X.X..........................................X.] duration: 619.841882ms FAIL (2025-08-07 23:54:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 619.860917ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......XXXXX..X............................................XX] duration: 1m0.000656916s FAIL (2025-08-07 23:54:00 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000678456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 11 out of 60 is failed (threshold=3) [....XXXXX.XX............................................XXXX] duration: 30.001119922s FAIL (2025-08-07 23:53:58 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== item: pb.BatchRequestItem_ObjectCommit Aug 7 23:45:42.664256383 (7h18m56.251312604s) [upload=7507]: Done uploading segments... Aug 7 23:45:42.664275208 (7h18m56.251331429s) [upload=7507]: Upload finished. err: <nil> Aug 7 23:46:08.906811448 (7h19m22.493867688s) [upload=7514]: Starting upload Aug 7 23:46:08.906885386 (7h19m22.493941627s) [upload=7514]: Done starting upload Aug 7 23:46:08.906964935 (7h19m22.494021175s) [upload=7514]: Uploading segments... Aug 7 23:46:08.906997987 (7h19m22.494054228s) [upload=7514]: Got next segment. Inline: true Aug 7 23:46:08.907003197 (7h19m22.494059437s) [upload=7514]: Waiting for error group managing segments... Aug 7 23:46:08.907046577 (7h19m22.494102807s) [upload=7514]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:08.907051767 (7h19m22.494108007s) [upload=7514]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:08.907055173 (7h19m22.494111414s) [upload=7514]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:08.965075092 (7h19m22.552131313s) [upload=7514]: Done uploading segments... Aug 7 23:46:08.965095970 (7h19m22.552152191s) [upload=7514]: Upload finished. err: <nil> Aug 7 23:47:00.017908282 (7h20m13.604964493s) [upload=7527]: Starting upload Aug 7 23:47:00.017955210 (7h20m13.605011411s) [upload=7527]: Done starting upload Aug 7 23:47:00.017998241 (7h20m13.605054452s) [upload=7527]: Uploading segments... Aug 7 23:47:00.018026292 (7h20m13.605082503s) [upload=7527]: Got next segment. Inline: true Aug 7 23:47:00.018029438 (7h20m13.60508565s) [upload=7527]: Waiting for error group managing segments... Aug 7 23:47:00.018058453 (7h20m13.605114664s) [upload=7527]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:47:00.018062390 (7h20m13.605118601s) [upload=7527]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:47:00.018064304 (7h20m13.605120515s) [upload=7527]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:00.075526210 (7h20m13.662582431s) [upload=7527]: Done uploading segments... Aug 7 23:47:00.075540917 (7h20m13.662597138s) [upload=7527]: Upload finished. err: <nil> Aug 7 23:48:09.882765252 (7h21m23.469821463s) [upload=7545]: Starting upload Aug 7 23:48:09.882860198 (7h21m23.46991642s) [upload=7545]: Done starting upload Aug 7 23:48:09.882935729 (7h21m23.469991941s) [upload=7545]: Uploading segments... Aug 7 23:48:09.882965635 (7h21m23.470021856s) [upload=7545]: Got next segment. Inline: true Aug 7 23:48:09.882969723 (7h21m23.470025944s) [upload=7545]: Waiting for error group managing segments... Aug 7 23:48:09.883002865 (7h21m23.470059086s) [upload=7545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:09.883006481 (7h21m23.470062702s) [upload=7545]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:09.883009045 (7h21m23.470065266s) [upload=7545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:09.951272229 (7h21m23.538328451s) [upload=7545]: Done uploading segments... Aug 7 23:48:09.951286926 (7h21m23.538343148s) [upload=7545]: Upload finished. err: <nil> Aug 7 23:48:58.772502770 (7h22m12.359559011s) [upload=7557]: Starting upload Aug 7 23:48:58.772574805 (7h22m12.359631046s) [upload=7557]: Done starting upload Aug 7 23:48:58.772645527 (7h22m12.359701767s) [upload=7557]: Uploading segments... Aug 7 23:48:58.772695300 (7h22m12.359751541s) [upload=7557]: Got next segment. Inline: true Aug 7 23:48:58.772701031 (7h22m12.359757271s) [upload=7557]: Waiting for error group managing segments... Aug 7 23:48:58.772770249 (7h22m12.359826501s) [upload=7557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:58.772780228 (7h22m12.359836469s) [upload=7557]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:58.772787010 (7h22m12.359843252s) [upload=7557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:58.840436652 (7h22m12.427492893s) [upload=7557]: Done uploading segments... Aug 7 23:48:58.840462661 (7h22m12.427518891s) [upload=7557]: Upload finished. err: <nil> Aug 7 23:50:01.971851413 (7h23m15.558907624s) [upload=7574]: Starting upload Aug 7 23:50:01.971890696 (7h23m15.558946907s) [upload=7574]: Done starting upload Aug 7 23:50:01.971977749 (7h23m15.559033961s) [upload=7574]: Uploading segments... Aug 7 23:50:01.972022943 (7h23m15.559079154s) [upload=7574]: Got next segment. Inline: true Aug 7 23:50:01.972026850 (7h23m15.559083062s) [upload=7574]: Waiting for error group managing segments... Aug 7 23:50:01.972060975 (7h23m15.559117185s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:01.972064511 (7h23m15.559120712s) [upload=7574]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:01.972066685 (7h23m15.559122896s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:02.063709030 (7h23m15.650765241s) [upload=7574]: Done uploading segments... Aug 7 23:50:02.063723547 (7h23m15.650779768s) [upload=7574]: Upload finished. err: <nil> Aug 7 23:50:41.160166556 (7h23m54.747222777s) [upload=7588]: Starting upload Aug 7 23:50:41.160220555 (7h23m54.747276777s) [upload=7588]: Done starting upload Aug 7 23:50:41.160264659 (7h23m54.74732088s) [upload=7588]: Uploading segments... Aug 7 23:50:41.160295956 (7h23m54.747352177s) [upload=7588]: Got next segment. Inline: true Aug 7 23:50:41.160299753 (7h23m54.747355964s) [upload=7588]: Waiting for error group managing segments... Aug 7 23:50:41.160331633 (7h23m54.747387844s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:41.160335861 (7h23m54.747392072s) [upload=7588]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:41.160338396 (7h23m54.747394617s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:41.222859735 (7h23m54.809915976s) [upload=7588]: Done uploading segments... Aug 7 23:50:41.222879913 (7h23m54.809936154s) [upload=7588]: Upload finished. err: <nil> Aug 7 23:51:17.131690986 (7h24m30.718747197s) [upload=7599]: Starting upload Aug 7 23:51:17.131738926 (7h24m30.718795147s) [upload=7599]: Done starting upload Aug 7 23:51:17.131792947 (7h24m30.718849178s) [upload=7599]: Uploading segments... Aug 7 23:51:17.131835896 (7h24m30.718892117s) [upload=7599]: Got next segment. Inline: true Aug 7 23:51:17.131842027 (7h24m30.718898239s) [upload=7599]: Waiting for error group managing segments... Aug 7 23:51:17.131883385 (7h24m30.718939627s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:17.131889146 (7h24m30.718945388s) [upload=7599]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:17.132006205 (7h24m30.719062435s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:38.787433326 (7h24m52.374489567s) [upload=7599]: Done uploading segments... Aug 7 23:51:38.787452212 (7h24m52.374508453s) [upload=7599]: Upload finished. err: <nil> Aug 7 23:52:00.256204647 (7h25m13.843260858s) [upload=7610]: Starting upload Aug 7 23:52:00.256263547 (7h25m13.843319758s) [upload=7610]: Done starting upload Aug 7 23:52:00.256316195 (7h25m13.843372406s) [upload=7610]: Uploading segments... Aug 7 23:52:00.256345671 (7h25m13.843401881s) [upload=7610]: Got next segment. Inline: true Aug 7 23:52:00.256349758 (7h25m13.843405969s) [upload=7610]: Waiting for error group managing segments... Aug 7 23:52:00.256399992 (7h25m13.843456203s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:00.256404780 (7h25m13.843460991s) [upload=7610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:00.256407806 (7h25m13.843464017s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:30.237600164 (7h25m43.824656416s) [upload=7610]: Done uploading segments... Aug 7 23:52:30.237630862 (7h25m43.824687113s) [upload=7610]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:53.146251246 (7h26m6.733307456s) [upload=7626]: Starting upload Aug 7 23:52:53.146307339 (7h26m6.733363561s) [upload=7626]: Done starting upload Aug 7 23:52:53.146350962 (7h26m6.733407173s) [upload=7626]: Uploading segments... Aug 7 23:52:53.146411344 (7h26m6.733467565s) [upload=7626]: Got next segment. Inline: true Aug 7 23:52:53.146415271 (7h26m6.733471493s) [upload=7626]: Waiting for error group managing segments... Aug 7 23:52:53.146449966 (7h26m6.733506177s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:53.146454174 (7h26m6.733510395s) [upload=7626]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:53.146456709 (7h26m6.73351293s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.132536521 (7h26m36.719592742s) [upload=7626]: Done uploading segments... Aug 7 23:53:23.132560907 (7h26m36.719617128s) [upload=7626]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:53:28.217132518 (7h26m41.804188759s) [upload=7638]: Starting upload Aug 7 23:53:28.217216185 (7h26m41.804272426s) [upload=7638]: Done starting upload Aug 7 23:53:28.217288009 (7h26m41.80434426s) [upload=7638]: Uploading segments... Aug 7 23:53:28.217336129 (7h26m41.80439237s) [upload=7638]: Got next segment. Inline: true Aug 7 23:53:28.217341890 (7h26m41.804398131s) [upload=7638]: Waiting for error group managing segments... Aug 7 23:53:28.217421438 (7h26m41.804477679s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:28.217427650 (7h26m41.804483901s) [upload=7638]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:28.217431617 (7h26m41.804487859s) [upload=7638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:58.204830989 (7h27m11.79188721s) [upload=7638]: Done uploading segments... Aug 7 23:53:58.204849294 (7h27m11.791905505s) [upload=7638]: Upload finished. err: metaclient: context deadline exceeded Duration:* 30.001141061s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.XX.X....X.X..........................................X] duration: 30.000347718s FAIL (2025-08-07 23:53:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000366883s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 458.21592ms FAIL (2025-08-07 23:53:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 458.243251ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.......X.XXXXX..X...........................................] duration: 215.127128ms FAIL (2025-08-07 23:53:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 215.148538ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X....XXX.X......................................X] duration: 30.000543878s FAIL (2025-08-07 23:53:32 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== [upload=7594]: Upload finished. err: <nil> Aug 7 23:52:06.728352003 (7h25m20.315408205s) [upload=7613]: Starting upload Aug 7 23:52:06.728452502 (7h25m20.315508713s) [upload=7613]: Done starting upload Aug 7 23:52:06.728510219 (7h25m20.31556647s) [upload=7613]: Uploading segments... Aug 7 23:52:06.728794470 (7h25m20.315850712s) [upload=7613]: Got next segment. Inline: false Aug 7 23:52:06.728839153 (7h25m20.315895395s) [upload=7613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:06.728845906 (7h25m20.315902147s) [upload=7613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:52:06.752678421 (7h25m20.339734662s) [seg_pos={0 0}, upload=7613]: Begin upload segment... Aug 7 23:52:06.752737542 (7h25m20.339793782s) [seg_pos={0 0}, upload=7613]: Done begin upload segment. Aug 7 23:52:06.752769762 (7h25m20.339826003s) [noise=true , address=77.90.31.12:4590 , piece_id=PQUGWXWL3BFAO2QOB3TN7ERFO7ZZXVYTEX2BJOT2Y5APPFPICJFQ, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.752823492 (7h25m20.339879713s) [noise=true , address=45.91.116.8:4045 , piece_id=JPMO37KLHKIQUAQBORBK64FXGHE4ZOVTGR4CEESADPJVIRIW2QFA, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.753236552 (7h25m20.340292804s) [noise=true , address=77.90.31.14:4661 , piece_id=LLFIZYQUTYSDQJ7EQBAEPJ3Z3S2FRBVEIIOV4B5IQYFVVBG4FZXA, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.753261680 (7h25m20.34031792s) [noise=true , address=104.36.82.4:4801 , piece_id=E5YH3CCHLGC3PZX6YXSVJ54YIWVMGL42IOSNOVFUWAKVVXKFTPUA, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.753565126 (7h25m20.340621337s) [noise=true , address=45.91.116.7:4010 , piece_id=36CQJILODTNMU66BQ6U7QEM6UZUTKAX2RHHAWKBHYU6ZJAXO4IGA, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.753655274 (7h25m20.340711516s) [noise=true , address=45.91.116.5:4299 , piece_id=XZKLTHUUSYNYNCQB7CZM3L2ZSDMZDCGGCO4NV4TPICMZM7BQKVGQ, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.753854125 (7h25m20.340910337s) [noise=true , address=104.36.82.7:4912 , piece_id=OUFQD5IYM422I32OHZA2XK5HGLX3IWO3QQIGB3OQPZ53ZMKU5KZA, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.754018442 (7h25m20.341074683s) [noise=true , address=45.91.116.9:4263 , piece_id=YWUCFNNUEOFPDOF3CD2DGASWYKY3OEHDR4G34VSXBN3UBPGQYVAQ, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.754133126 (7h25m20.341189337s) [noise=true , address=77.90.31.18:4806 , piece_id=UTIDGJ6FJSQSM4NMZMM6EIAOXXQGDWKR5FTEUCZZM4JUDL4H7IBQ, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.754397169 (7h25m20.341453411s) [noise=true , address=45.91.116.6:4975 , piece_id=RFDVEUP3YNCDZCKGGBFNVLQ2RAVRBF5ZR2MH6NZDINNHLMVC34FA, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.754462500 (7h25m20.341518721s) [noise=true , address=45.91.116.10:4081 , piece_id=2VCAS4RZ645LBNNB3CRUI6BIWKMT7DANB25XFN3PN2S3OPBR5BMA, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.754745610 (7h25m20.341801821s) [noise=true , address=104.36.82.9:4007 , piece_id=IYNBWMWGJJHXWCFY2UNTN6BOZUZQCEG5HMDP65AVBNJZXBDL25CA, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.754756841 (7h25m20.341813082s) [noise=true , address=104.36.82.4:4803 , piece_id=APSBYVSPJ3BLZ36FPEZQF3NO2MAQRGCBIOT24KAA47I5N2O3ZI2Q, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.755031362 (7h25m20.342087574s) [noise=true , address=45.91.116.11:4119 , piece_id=JVSG7QVEL5PFH25WQDGUROYOLK256J7GCEMA4TOSSVXIVI3ZNUYA, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.755131800 (7h25m20.342188041s) [noise=true , address=45.91.116.4:4938 , piece_id=YNZBEVYM5S7VV54MSNF26HMGU7JFHTPA5TIOI42C5ZCOZUDGS3SA, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.755569387 (7h25m20.342625638s) [noise=true , address=45.91.116.13:4153 , piece_id=77TRA6PWDU6D5NTRN27BC2PEUNRPIW7ONITHPM56QQ4HWYZ5V5NA, seg_pos={0 0}, upload=7613]: Uploading piece... Aug 7 23:52:06.841187318 (7h25m20.42824353s) [upload=7613]: Next returned nil segment Aug 7 23:52:06.841207075 (7h25m20.428263286s) [upload=7613]: Waiting for error group managing segments... Aug 7 23:52:06.849202168 (7h25m20.43625841s) [noise=true , address=45.91.116.4:4938 , piece_id=YNZBEVYM5S7VV54MSNF26HMGU7JFHTPA5TIOI42C5ZCOZUDGS3SA, seg_pos={0 0}, upload=7613]: Done uploading piece. err: <nil> Aug 7 23:52:06.849781229 (7h25m20.436837439s) [noise=true , address=77.90.31.12:4590 , piece_id=PQUGWXWL3BFAO2QOB3TN7ERFO7ZZXVYTEX2BJOT2Y5APPFPICJFQ, seg_pos={0 0}, upload=7613]: Done uploading piece. err: <nil> Aug 7 23:52:06.850187878 (7h25m20.43724409s) [noise=true , address=45.91.116.10:4081 , piece_id=2VCAS4RZ645LBNNB3CRUI6BIWKMT7DANB25XFN3PN2S3OPBR5BMA, seg_pos={0 0}, upload=7613]: Done uploading piece. err: <nil> Aug 7 23:52:06.850510370 (7h25m20.437566621s) [noise=true , address=45.91.116.13:4153 , piece_id=77TRA6PWDU6D5NTRN27BC2PEUNRPIW7ONITHPM56QQ4HWYZ5V5NA, seg_pos={0 0}, upload=7613]: Done uploading piece. err: <nil> Aug 7 23:52:06.850948047 (7h25m20.438004258s) [noise=true , address=45.91.116.9:4263 , piece_id=YWUCFNNUEOFPDOF3CD2DGASWYKY3OEHDR4G34VSXBN3UBPGQYVAQ, seg_pos={0 0}, upload=7613]: Done uploading piece. err: <nil> Aug 7 23:52:06.851335932 (7h25m20.438392142s) [noise=true , address=45.91.116.8:4045 , piece_id=JPMO37KLHKIQUAQBORBK64FXGHE4ZOVTGR4CEESADPJVIRIW2QFA, seg_pos={0 0}, upload=7613]: Done uploading piece. err: <nil> Aug 7 23:52:06.851618539 (7h25m20.438674749s) [noise=true , address=45.91.116.11:4119 , piece_id=JVSG7QVEL5PFH25WQDGUROYOLK256J7GCEMA4TOSSVXIVI3ZNUYA, seg_pos={0 0}, upload=7613]: Done uploading piece. err: <nil> Aug 7 23:52:06.851736418 (7h25m20.43879267s) [noise=true , address=45.91.116.7:4010 , piece_id=36CQJILODTNMU66BQ6U7QEM6UZUTKAX2RHHAWKBHYU6ZJAXO4IGA, seg_pos={0 0}, upload=7613]: Done uploading piece. err: <nil> Aug 7 23:52:06.851936792 (7h25m20.438993003s) [noise=true , address=77.90.31.18:4806 , piece_id=UTIDGJ6FJSQSM4NMZMM6EIAOXXQGDWKR5FTEUCZZM4JUDL4H7IBQ, seg_pos={0 0}, upload=7613]: Done uploading piece. err: <nil> Aug 7 23:52:06.853958856 (7h25m20.441015098s) [noise=true , address=45.91.116.5:4299 , piece_id=XZKLTHUUSYNYNCQB7CZM3L2ZSDMZDCGGCO4NV4TPICMZM7BQKVGQ, seg_pos={0 0}, upload=7613]: Done uploading piece. err: <nil> Aug 7 23:52:06.854659484 (7h25m20.441715696s) [noise=true , address=45.91.116.6:4975 , piece_id=RFDVEUP3YNCDZCKGGBFNVLQ2RAVRBF5ZR2MH6NZDINNHLMVC34FA, seg_pos={0 0}, upload=7613]: Done uploading piece. err: <nil> Aug 7 23:52:06.863188123 (7h25m20.450244334s) [noise=true , address=104.36.82.7:4912 , piece_id=OUFQD5IYM422I32OHZA2XK5HGLX3IWO3QQIGB3OQPZ53ZMKU5KZA, seg_pos={0 0}, upload=7613]: Done uploading piece. err: <nil> Aug 7 23:52:06.864977654 (7h25m20.452033895s) [noise=true , address=104.36.82.4:4803 , piece_id=APSBYVSPJ3BLZ36FPEZQF3NO2MAQRGCBIOT24KAA47I5N2O3ZI2Q, seg_pos={0 0}, upload=7613]: Done uploading piece. err: <nil> Aug 7 23:52:06.864998413 (7h25m20.452054654s) [seg_pos={0 0}, upload=7613]: Segment reached optimal threshold of 13 pieces. Aug 7 23:52:06.865176945 (7h25m20.452233157s) [noise=true , address=77.90.31.14:4661 , piece_id=LLFIZYQUTYSDQJ7EQBAEPJ3Z3S2FRBVEIIOV4B5IQYFVVBG4FZXA, seg_pos={0 0}, upload=7613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:06.865299825 (7h25m20.452356066s) [noise=true , address=104.36.82.4:4801 , piece_id=E5YH3CCHLGC3PZX6YXSVJ54YIWVMGL42IOSNOVFUWAKVVXKFTPUA, seg_pos={0 0}, upload=7613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:06.865339630 (7h25m20.45239584s) [noise=true , address=104.36.82.9:4007 , piece_id=IYNBWMWGJJHXWCFY2UNTN6BOZUZQCEG5HMDP65AVBNJZXBDL25CA, seg_pos={0 0}, upload=7613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:8386->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:8386->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:8386->104.36.82.9:4007: use of closed network connection Aug 7 23:52:06.865360338 (7h25m20.45241654s) [seg_pos={0 0}, upload=7613]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:52:06.865450757 (7h25m20.452506968s) [upload=7613]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:52:06.865454935 (7h25m20.452511146s) [upload=7613]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:07.015091480 (7h25m20.602147711s) [upload=7613]: Done uploading segments... Aug 7 23:52:07.015111889 (7h25m20.60216813s) [upload=7613]: Upload finished. err: <nil> Aug 7 23:53:02.405645052 (7h26m15.992701273s) [upload=7627]: Starting upload Aug 7 23:53:02.405704283 (7h26m15.992760504s) [upload=7627]: Done starting upload Aug 7 23:53:02.405806533 (7h26m15.992862744s) [upload=7627]: Uploading segments... Aug 7 23:53:02.406021685 (7h26m15.993077906s) [upload=7627]: Got next segment. Inline: false Aug 7 23:53:02.406061369 (7h26m15.99311764s) [upload=7627]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:02.406068102 (7h26m15.993124323s) [upload=7627]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:32.387466061 (7h26m45.974522302s) [upload=7627]: Next segment err: context deadline exceeded Aug 7 23:53:32.387608347 (7h26m45.974664588s) [upload=7627]: Done uploading segments... Aug 7 23:53:32.387634907 (7h26m45.974691158s) [upload=7627]: Upload finished. err: context deadline exceeded Duration: 30.000638194s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 10 out of 60 is failed (threshold=3) [.....XXXXX.XX............................................XXX] duration: 30.000953405s FAIL (2025-08-07 23:53:23 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [upload=7488]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:44:35.894656303 (7h17m49.481712545s) [upload=7488]: Done uploading segments... Aug 7 23:44:35.894676211 (7h17m49.481732452s) [upload=7488]: Upload finished. err: <nil> Aug 7 23:45:42.609270547 (7h18m56.196326759s) [upload=7507]: Starting upload Aug 7 23:45:42.609334396 (7h18m56.196390617s) [upload=7507]: Done starting upload Aug 7 23:45:42.609380774 (7h18m56.196436984s) [upload=7507]: Uploading segments... Aug 7 23:45:42.609412683 (7h18m56.196468914s) [upload=7507]: Got next segment. Inline: true Aug 7 23:45:42.609418174 (7h18m56.196474385s) [upload=7507]: Waiting for error group managing segments... Aug 7 23:45:42.609455492 (7h18m56.196511713s) [upload=7507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:45:42.609459600 (7h18m56.196515811s) [upload=7507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:45:42.609462095 (7h18m56.196518316s) [upload=7507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:45:42.664256383 (7h18m56.251312604s) [upload=7507]: Done uploading segments... Aug 7 23:45:42.664275208 (7h18m56.251331429s) [upload=7507]: Upload finished. err: <nil> Aug 7 23:46:08.906811448 (7h19m22.493867688s) [upload=7514]: Starting upload Aug 7 23:46:08.906885386 (7h19m22.493941627s) [upload=7514]: Done starting upload Aug 7 23:46:08.906964935 (7h19m22.494021175s) [upload=7514]: Uploading segments... Aug 7 23:46:08.906997987 (7h19m22.494054228s) [upload=7514]: Got next segment. Inline: true Aug 7 23:46:08.907003197 (7h19m22.494059437s) [upload=7514]: Waiting for error group managing segments... Aug 7 23:46:08.907046577 (7h19m22.494102807s) [upload=7514]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:08.907051767 (7h19m22.494108007s) [upload=7514]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:08.907055173 (7h19m22.494111414s) [upload=7514]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:08.965075092 (7h19m22.552131313s) [upload=7514]: Done uploading segments... Aug 7 23:46:08.965095970 (7h19m22.552152191s) [upload=7514]: Upload finished. err: <nil> Aug 7 23:47:00.017908282 (7h20m13.604964493s) [upload=7527]: Starting upload Aug 7 23:47:00.017955210 (7h20m13.605011411s) [upload=7527]: Done starting upload Aug 7 23:47:00.017998241 (7h20m13.605054452s) [upload=7527]: Uploading segments... Aug 7 23:47:00.018026292 (7h20m13.605082503s) [upload=7527]: Got next segment. Inline: true Aug 7 23:47:00.018029438 (7h20m13.60508565s) [upload=7527]: Waiting for error group managing segments... Aug 7 23:47:00.018058453 (7h20m13.605114664s) [upload=7527]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:47:00.018062390 (7h20m13.605118601s) [upload=7527]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:47:00.018064304 (7h20m13.605120515s) [upload=7527]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:00.075526210 (7h20m13.662582431s) [upload=7527]: Done uploading segments... Aug 7 23:47:00.075540917 (7h20m13.662597138s) [upload=7527]: Upload finished. err: <nil> Aug 7 23:48:09.882765252 (7h21m23.469821463s) [upload=7545]: Starting upload Aug 7 23:48:09.882860198 (7h21m23.46991642s) [upload=7545]: Done starting upload Aug 7 23:48:09.882935729 (7h21m23.469991941s) [upload=7545]: Uploading segments... Aug 7 23:48:09.882965635 (7h21m23.470021856s) [upload=7545]: Got next segment. Inline: true Aug 7 23:48:09.882969723 (7h21m23.470025944s) [upload=7545]: Waiting for error group managing segments... Aug 7 23:48:09.883002865 (7h21m23.470059086s) [upload=7545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:09.883006481 (7h21m23.470062702s) [upload=7545]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:09.883009045 (7h21m23.470065266s) [upload=7545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:09.951272229 (7h21m23.538328451s) [upload=7545]: Done uploading segments... Aug 7 23:48:09.951286926 (7h21m23.538343148s) [upload=7545]: Upload finished. err: <nil> Aug 7 23:48:58.772502770 (7h22m12.359559011s) [upload=7557]: Starting upload Aug 7 23:48:58.772574805 (7h22m12.359631046s) [upload=7557]: Done starting upload Aug 7 23:48:58.772645527 (7h22m12.359701767s) [upload=7557]: Uploading segments... Aug 7 23:48:58.772695300 (7h22m12.359751541s) [upload=7557]: Got next segment. Inline: true Aug 7 23:48:58.772701031 (7h22m12.359757271s) [upload=7557]: Waiting for error group managing segments... Aug 7 23:48:58.772770249 (7h22m12.359826501s) [upload=7557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:58.772780228 (7h22m12.359836469s) [upload=7557]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:58.772787010 (7h22m12.359843252s) [upload=7557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:58.840436652 (7h22m12.427492893s) [upload=7557]: Done uploading segments... Aug 7 23:48:58.840462661 (7h22m12.427518891s) [upload=7557]: Upload finished. err: <nil> Aug 7 23:50:01.971851413 (7h23m15.558907624s) [upload=7574]: Starting upload Aug 7 23:50:01.971890696 (7h23m15.558946907s) [upload=7574]: Done starting upload Aug 7 23:50:01.971977749 (7h23m15.559033961s) [upload=7574]: Uploading segments... Aug 7 23:50:01.972022943 (7h23m15.559079154s) [upload=7574]: Got next segment. Inline: true Aug 7 23:50:01.972026850 (7h23m15.559083062s) [upload=7574]: Waiting for error group managing segments... Aug 7 23:50:01.972060975 (7h23m15.559117185s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:01.972064511 (7h23m15.559120712s) [upload=7574]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:01.972066685 (7h23m15.559122896s) [upload=7574]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:02.063709030 (7h23m15.650765241s) [upload=7574]: Done uploading segments... Aug 7 23:50:02.063723547 (7h23m15.650779768s) [upload=7574]: Upload finished. err: <nil> Aug 7 23:50:41.160166556 (7h23m54.747222777s) [upload=7588]: Starting upload Aug 7 23:50:41.160220555 (7h23m54.747276777s) [upload=7588]: Done starting upload Aug 7 23:50:41.160264659 (7h23m54.74732088s) [upload=7588]: Uploading segments... Aug 7 23:50:41.160295956 (7h23m54.747352177s) [upload=7588]: Got next segment. Inline: true Aug 7 23:50:41.160299753 (7h23m54.747355964s) [upload=7588]: Waiting for error group managing segments... Aug 7 23:50:41.160331633 (7h23m54.747387844s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:41.160335861 (7h23m54.747392072s) [upload=7588]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:41.160338396 (7h23m54.747394617s) [upload=7588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:41.222859735 (7h23m54.809915976s) [upload=7588]: Done uploading segments... Aug 7 23:50:41.222879913 (7h23m54.809936154s) [upload=7588]: Upload finished. err: <nil> Aug 7 23:51:17.131690986 (7h24m30.718747197s) [upload=7599]: Starting upload Aug 7 23:51:17.131738926 (7h24m30.718795147s) [upload=7599]: Done starting upload Aug 7 23:51:17.131792947 (7h24m30.718849178s) [upload=7599]: Uploading segments... Aug 7 23:51:17.131835896 (7h24m30.718892117s) [upload=7599]: Got next segment. Inline: true Aug 7 23:51:17.131842027 (7h24m30.718898239s) [upload=7599]: Waiting for error group managing segments... Aug 7 23:51:17.131883385 (7h24m30.718939627s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:17.131889146 (7h24m30.718945388s) [upload=7599]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:17.132006205 (7h24m30.719062435s) [upload=7599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:38.787433326 (7h24m52.374489567s) [upload=7599]: Done uploading segments... Aug 7 23:51:38.787452212 (7h24m52.374508453s) [upload=7599]: Upload finished. err: <nil> Aug 7 23:52:00.256204647 (7h25m13.843260858s) [upload=7610]: Starting upload Aug 7 23:52:00.256263547 (7h25m13.843319758s) [upload=7610]: Done starting upload Aug 7 23:52:00.256316195 (7h25m13.843372406s) [upload=7610]: Uploading segments... Aug 7 23:52:00.256345671 (7h25m13.843401881s) [upload=7610]: Got next segment. Inline: true Aug 7 23:52:00.256349758 (7h25m13.843405969s) [upload=7610]: Waiting for error group managing segments... Aug 7 23:52:00.256399992 (7h25m13.843456203s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:00.256404780 (7h25m13.843460991s) [upload=7610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:00.256407806 (7h25m13.843464017s) [upload=7610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:30.237600164 (7h25m43.824656416s) [upload=7610]: Done uploading segments... Aug 7 23:52:30.237630862 (7h25m43.824687113s) [upload=7610]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:53.146251246 (7h26m6.733307456s) [upload=7626]: Starting upload Aug 7 23:52:53.146307339 (7h26m6.733363561s) [upload=7626]: Done starting upload Aug 7 23:52:53.146350962 (7h26m6.733407173s) [upload=7626]: Uploading segments... Aug 7 23:52:53.146411344 (7h26m6.733467565s) [upload=7626]: Got next segment. Inline: true Aug 7 23:52:53.146415271 (7h26m6.733471493s) [upload=7626]: Waiting for error group managing segments... Aug 7 23:52:53.146449966 (7h26m6.733506177s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:53.146454174 (7h26m6.733510395s) [upload=7626]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:53.146456709 (7h26m6.73351293s) [upload=7626]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.132536521 (7h26m36.719592742s) [upload=7626]: Done uploading segments... Aug 7 23:53:23.132560907 (7h26m36.719617128s) [upload=7626]: Upload finished. err: metaclient: context deadline exceeded Duration:* 30.000969967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 173.725377ms FAIL (2025-08-07 23:53:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ne Aug 7 23:09:39.944386324 (6h42m53.531442545s) [upload=6887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:40.134449947 (6h42m53.721506158s) [upload=6887]: Done uploading segments... Aug 7 23:09:40.134464625 (6h42m53.721520846s) [upload=6887]: Upload finished. err: <nil> Aug 7 23:10:38.125235482 (6h43m51.712291723s) [upload=6908]: Starting upload Aug 7 23:10:38.125321953 (6h43m51.712378205s) [upload=6908]: Done starting upload Aug 7 23:10:38.125506868 (6h43m51.712563119s) [upload=6908]: Uploading segments... Aug 7 23:10:38.125577140 (6h43m51.712633391s) [upload=6908]: Got next segment. Inline: true Aug 7 23:10:38.125584223 (6h43m51.712640464s) [upload=6908]: Waiting for error group managing segments... Aug 7 23:10:38.125639727 (6h43m51.712695977s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:38.125651738 (6h43m51.71270799s) [upload=6908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:38.125657910 (6h43m51.712714161s) [upload=6908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:38.249068118 (6h43m51.836124368s) [upload=6908]: Done uploading segments... Aug 7 23:10:38.249095648 (6h43m51.836151889s) [upload=6908]: Upload finished. err: <nil> Aug 7 23:11:37.996040842 (6h44m51.583097123s) [upload=6923]: Starting upload Aug 7 23:11:37.996093240 (6h44m51.583149461s) [upload=6923]: Done starting upload Aug 7 23:11:37.996137332 (6h44m51.583193553s) [upload=6923]: Uploading segments... Aug 7 23:11:37.996236618 (6h44m51.58329286s) [upload=6923]: Got next segment. Inline: true Aug 7 23:11:37.996246336 (6h44m51.583302557s) [upload=6923]: Waiting for error group managing segments... Aug 7 23:11:37.996301449 (6h44m51.58335767s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:37.996308312 (6h44m51.583364553s) [upload=6923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:37.996313571 (6h44m51.583369783s) [upload=6923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:38.115493220 (6h44m51.702549431s) [upload=6923]: Done uploading segments... Aug 7 23:11:38.115513639 (6h44m51.702569849s) [upload=6923]: Upload finished. err: <nil> Aug 7 23:12:37.449808013 (6h45m51.036864224s) [upload=6941]: Starting upload Aug 7 23:12:37.449858507 (6h45m51.036914718s) [upload=6941]: Done starting upload Aug 7 23:12:37.449904994 (6h45m51.036961205s) [upload=6941]: Uploading segments... Aug 7 23:12:37.449949496 (6h45m51.037005707s) [upload=6941]: Got next segment. Inline: true Aug 7 23:12:37.449955127 (6h45m51.037011338s) [upload=6941]: Waiting for error group managing segments... Aug 7 23:12:37.449995823 (6h45m51.037052034s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.450001143 (6h45m51.037057354s) [upload=6941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.450004189 (6h45m51.0370604s) [upload=6941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.534556583 (6h45m51.121612794s) [upload=6941]: Done uploading segments... Aug 7 23:12:37.534575238 (6h45m51.121631449s) [upload=6941]: Upload finished. err: <nil> Aug 7 23:13:12.303607944 (6h46m25.890664155s) [upload=6950]: Starting upload Aug 7 23:13:12.303650153 (6h46m25.890706374s) [upload=6950]: Done starting upload Aug 7 23:13:12.303719743 (6h46m25.890775964s) [upload=6950]: Uploading segments... Aug 7 23:13:12.303750360 (6h46m25.890806581s) [upload=6950]: Got next segment. Inline: true Aug 7 23:13:12.303754137 (6h46m25.890810358s) [upload=6950]: Waiting for error group managing segments... Aug 7 23:13:12.303786066 (6h46m25.890842278s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:12.303789663 (6h46m25.890845884s) [upload=6950]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:12.303792177 (6h46m25.890848398s) [upload=6950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:12.374674527 (6h46m25.961730738s) [upload=6950]: Done uploading segments... Aug 7 23:13:12.374688663 (6h46m25.961744874s) [upload=6950]: Upload finished. err: <nil> Aug 7 23:13:56.776579420 (6h47m10.36363566s) [upload=6962]: Starting upload Aug 7 23:13:56.776644211 (6h47m10.363700462s) [upload=6962]: Done starting upload Aug 7 23:13:56.776723829 (6h47m10.363780071s) [upload=6962]: Uploading segments... Aug 7 23:13:56.776773401 (6h47m10.363829643s) [upload=6962]: Got next segment. Inline: true Aug 7 23:13:56.776779252 (6h47m10.363835494s) [upload=6962]: Waiting for error group managing segments... Aug 7 23:13:56.776832092 (6h47m10.363888332s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.776839315 (6h47m10.363895556s) [upload=6962]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.776843042 (6h47m10.363899293s) [upload=6962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.833253067 (6h47m10.420309309s) [upload=6962]: Done uploading segments... Aug 7 23:13:56.833272594 (6h47m10.420328835s) [upload=6962]: Upload finished. err: <nil> Aug 7 23:14:23.847990399 (6h47m37.43504664s) [upload=6971]: Starting upload Aug 7 23:14:23.848064437 (6h47m37.435120678s) [upload=6971]: Done starting upload Aug 7 23:14:23.848133346 (6h47m37.435189557s) [upload=6971]: Uploading segments... Aug 7 23:14:23.848226790 (6h47m37.435283002s) [upload=6971]: Got next segment. Inline: true Aug 7 23:14:23.848231990 (6h47m37.435288201s) [upload=6971]: Waiting for error group managing segments... Aug 7 23:14:23.848270613 (6h47m37.435326823s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:23.848275903 (6h47m37.435332124s) [upload=6971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:23.848278567 (6h47m37.435334779s) [upload=6971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:24.081872322 (6h47m37.668928543s) [upload=6971]: Done uploading segments... Aug 7 23:14:24.081896036 (6h47m37.668952257s) [upload=6971]: Upload finished. err: <nil> Aug 7 23:14:49.940481761 (6h48m3.527538001s) [upload=6979]: Starting upload Aug 7 23:14:49.940552292 (6h48m3.527608533s) [upload=6979]: Done starting upload Aug 7 23:14:49.940622082 (6h48m3.527678324s) [upload=6979]: Uploading segments... Aug 7 23:14:49.940755703 (6h48m3.527811943s) [upload=6979]: Got next segment. Inline: true Aug 7 23:14:49.940763758 (6h48m3.527819998s) [upload=6979]: Waiting for error group managing segments... Aug 7 23:14:49.940819711 (6h48m3.527875952s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:49.940826003 (6h48m3.527882244s) [upload=6979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:49.940829760 (6h48m3.527886001s) [upload=6979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:50.031323214 (6h48m3.618379435s) [upload=6979]: Done uploading segments... Aug 7 23:14:50.031337651 (6h48m3.618393872s) [upload=6979]: Upload finished. err: <nil> Aug 7 23:15:44.466066460 (6h48m58.053122681s) [upload=6995]: Starting upload Aug 7 23:15:44.466127083 (6h48m58.053183304s) [upload=6995]: Done starting upload Aug 7 23:15:44.466180152 (6h48m58.053236374s) [upload=6995]: Uploading segments... Aug 7 23:15:44.466238350 (6h48m58.053294571s) [upload=6995]: Got next segment. Inline: true Aug 7 23:15:44.466243571 (6h48m58.053299781s) [upload=6995]: Waiting for error group managing segments... Aug 7 23:15:44.466279948 (6h48m58.05333617s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:44.466284317 (6h48m58.053340538s) [upload=6995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:44.466286601 (6h48m58.053342812s) [upload=6995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:44.537462409 (6h48m58.124518621s) [upload=6995]: Done uploading segments... Aug 7 23:15:44.537479490 (6h48m58.124535701s) [upload=6995]: Upload finished. err: <nil> Aug 7 23:16:06.153565692 (6h49m19.740621913s) [upload=6998]: Starting upload Aug 7 23:16:06.153632095 (6h49m19.740688306s) [upload=6998]: Done starting upload Aug 7 23:16:06.153689062 (6h49m19.740745293s) [upload=6998]: Uploading segments... Aug 7 23:16:06.153727503 (6h49m19.740783714s) [upload=6998]: Got next segment. Inline: true Aug 7 23:16:06.153733284 (6h49m19.740789505s) [upload=6998]: Waiting for error group managing segments... Aug 7 23:16:06.153775644 (6h49m19.740831865s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:06.153781144 (6h49m19.740837365s) [upload=6998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:06.153785161 (6h49m19.740841372s) [upload=6998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:06.348027973 (6h49m19.935084194s) [upload=6998]: Done uploading segments... Aug 7 23:16:06.348046217 (6h49m19.935102438s) [upload=6998]: Upload finished. err: <nil> Aug 7 23:16:30.802675866 (6h49m44.389732087s) [upload=7006]: Starting upload Aug 7 23:16:30.802719959 (6h49m44.38977618s) [upload=7006]: Done starting upload Aug 7 23:16:30.802764312 (6h49m44.389820533s) [upload=7006]: Uploading segments... Aug 7 23:16:30.802792133 (6h49m44.389848344s) [upload=7006]: Got next segment. Inline: true Aug 7 23:16:30.802795820 (6h49m44.389852031s) [upload=7006]: Waiting for error group managing segments... Aug 7 23:16:30.802827569 (6h49m44.38988378s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.802831226 (6h49m44.389887437s) [upload=7006]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.802833681 (6h49m44.389889902s) [upload=7006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.805506480 (6h49m44.392562691s) [upload=7006]: Done uploading segments... Aug 7 23:16:30.805550723 (6h49m44.392606934s) [upload=7006]: Upload finished. err: metaclient: Too Many Requests Duration:* 173.749432ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........XXX.X.X.........................................X] duration: 1m0.000829858s FAIL (2025-08-07 23:53:11 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== QBRGXTTEOU4SJY2SWUTFQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.346810570 (7h24m30.933866781s) [noise=true , address=104.36.82.42:4801 , piece_id=ILNDV4CZCPEST6R4DUTSHQ5OJS2N4BLPRLDYIYNBTZKRR4RSCOFA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.346916297 (7h24m30.933972519s) [noise=true , address=104.36.82.59:4327 , piece_id=5AM7TVDV5ORD5WPHKEHOM652JETCJ6VH3JOJ6HPJTIOTABTYVELA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347112754 (7h24m30.934168975s) [noise=true , address=104.36.82.44:4904 , piece_id=TFXFZ53DGPOVVXTTZMRMTJR53DGXNHWUF43I7OCC4YNCRW4QUCKA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347141428 (7h24m30.934197638s) [noise=true , address=79.127.201.215:4343 , piece_id=6IVWBNSJNI4ZAEIWFTRYUHIHZ35YTTZF3FTEF7G76YEBUP2NBJFA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347399839 (7h24m30.934456051s) [noise=true , address=104.36.82.35:4611 , piece_id=FE47XNPAFZAELTDVIO6C4URDMXRWEXCM37BFM4LANQQ6QDSU7HAQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347417763 (7h24m30.934473974s) [noise=true , address=104.36.82.4:4806 , piece_id=X42UHVJVQMP46Y7QDE377JH6FNETULCU4JBDSGTVJDRSUE5LAVOA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347625520 (7h24m30.934681731s) [noise=true , address=45.91.116.10:4098 , piece_id=5ITNSAGAAD2QAEAZGQSRYU2GSIS54FPDHU6HETL6QVR57KUDYICQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347755754 (7h24m30.934811985s) [noise=true , address=104.36.82.45:4921 , piece_id=GQXCWGC6EHWZONLT426WVNPLASR7SECUANPQGGZNMASCRRD4X4SQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.347968761 (7h24m30.935024982s) [noise=true , address=104.36.82.62:4438 , piece_id=3ND3RVOIIEJMN5GH7KZRHPYVNIH5KB62OU6KEV5DVOLCAZI4BQLA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348133560 (7h24m30.935189781s) [noise=true , address=104.36.82.39:4729 , piece_id=B7IGTQUWZAQNQDUYOUAB3WGYK5QHKXXJKRDGKMEGDK6KS4ZFC5GA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348193100 (7h24m30.935249311s) [noise=true , address=104.36.82.23:4475 , piece_id=LZMIGPR677ZHFYTXZJAMLW3GC3XLON7LGPRRBWZWVEXG4GGLEEEA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348536501 (7h24m30.935592722s) [noise=true , address=104.36.82.61:4406 , piece_id=N63ZWSHAILE55ZOUL5H6RYDUTB5F2F3EWMDT52RPJ5F46MLKRDMA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348656385 (7h24m30.935712596s) [noise=true , address=104.36.82.12:4178 , piece_id=NQWZGYOLKX6SHV3RXIXVGFDAXFP4VMFMP5FJKQPZ63YZ2NK3RJDQ, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348854374 (7h24m30.935910585s) [noise=true , address=104.36.82.15:4288 , piece_id=KWW7E4ZMO42C3YEAH3WBCEYSW6TTNDX7TQ4MEITLSC2RR2CE4VDA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.348919236 (7h24m30.935975457s) [noise=true , address=104.36.82.36:4649 , piece_id=NV2V22M4HHIEHH4RENXMFHFYXHQF7LPJRPPD7ZBKHPSZFKI2BVTA, seg_pos={0 0}, upload=7600]: Uploading piece... Aug 7 23:51:17.430583697 (7h24m31.017639918s) [upload=7600]: Next returned nil segment Aug 7 23:51:17.430596350 (7h24m31.017652571s) [upload=7600]: Waiting for error group managing segments... Aug 7 23:51:17.450496095 (7h24m31.037552306s) [noise=true , address=104.36.82.39:4729 , piece_id=B7IGTQUWZAQNQDUYOUAB3WGYK5QHKXXJKRDGKMEGDK6KS4ZFC5GA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.457273115 (7h24m31.044329336s) [noise=true , address=45.91.116.11:4121 , piece_id=UI4HOUG4KLWC3LFGDVZ3C3IUMB2DGJP2QOTIOLWS5C2HUAMSATMA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.458658320 (7h24m31.045714532s) [noise=true , address=104.36.82.36:4649 , piece_id=NV2V22M4HHIEHH4RENXMFHFYXHQF7LPJRPPD7ZBKHPSZFKI2BVTA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.464324767 (7h24m31.051380978s) [noise=true , address=104.36.82.62:4438 , piece_id=3ND3RVOIIEJMN5GH7KZRHPYVNIH5KB62OU6KEV5DVOLCAZI4BQLA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.469258046 (7h24m31.056314256s) [noise=true , address=104.36.82.4:4806 , piece_id=X42UHVJVQMP46Y7QDE377JH6FNETULCU4JBDSGTVJDRSUE5LAVOA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.470547041 (7h24m31.057603252s) [noise=true , address=45.91.116.10:4098 , piece_id=5ITNSAGAAD2QAEAZGQSRYU2GSIS54FPDHU6HETL6QVR57KUDYICQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.482895480 (7h24m31.069951691s) [noise=true , address=104.36.82.59:4327 , piece_id=5AM7TVDV5ORD5WPHKEHOM652JETCJ6VH3JOJ6HPJTIOTABTYVELA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.503848269 (7h24m31.090904481s) [noise=true , address=104.36.82.61:4406 , piece_id=N63ZWSHAILE55ZOUL5H6RYDUTB5F2F3EWMDT52RPJ5F46MLKRDMA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.589438619 (7h24m31.17649484s) [noise=true , address=79.127.201.215:4343 , piece_id=6IVWBNSJNI4ZAEIWFTRYUHIHZ35YTTZF3FTEF7G76YEBUP2NBJFA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.633863581 (7h24m31.220919802s) [noise=true , address=104.36.82.42:4801 , piece_id=ILNDV4CZCPEST6R4DUTSHQ5OJS2N4BLPRLDYIYNBTZKRR4RSCOFA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.636493760 (7h24m31.22354997s) [noise=true , address=104.36.82.44:4904 , piece_id=TFXFZ53DGPOVVXTTZMRMTJR53DGXNHWUF43I7OCC4YNCRW4QUCKA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.637613389 (7h24m31.224669611s) [noise=true , address=104.36.82.23:4475 , piece_id=LZMIGPR677ZHFYTXZJAMLW3GC3XLON7LGPRRBWZWVEXG4GGLEEEA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.654845844 (7h24m31.241902065s) [noise=true , address=104.36.82.22:4524 , piece_id=66YIUJKSDV52T5NMMFHYCZH7L34UKH3RLIS5WZ3VWRBL3KEE7UWQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.853873712 (7h24m31.440929933s) [noise=true , address=104.36.82.12:4178 , piece_id=NQWZGYOLKX6SHV3RXIXVGFDAXFP4VMFMP5FJKQPZ63YZ2NK3RJDQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: <nil> Aug 7 23:51:17.853888850 (7h24m31.440945061s) [seg_pos={0 0}, upload=7600]: Segment reached optimal threshold of 14 pieces. Aug 7 23:51:17.853987013 (7h24m31.441043234s) [noise=true , address=104.36.82.45:4921 , piece_id=GQXCWGC6EHWZONLT426WVNPLASR7SECUANPQGGZNMASCRRD4X4SQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bBRKXiuSN2cw6KrdP8q4Dsf4MiFCUSgMEGjqG2dKL9RLTQra1): protocol: expected piece hash; context canceled; EOF Aug 7 23:51:17.854295209 (7h24m31.44135142s) [noise=true , address=104.36.82.46:4499 , piece_id=6OBTEQJKT4AJRGUT27EYWNLG4E2TKBCQBRGXTTEOU4SJY2SWUTFQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc): protocol: write tcp 5.161.89.252:63506->104.36.82.46:4499: use of closed network connection; write tcp 5.161.89.252:63506->104.36.82.46:4499: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63506->104.36.82.46:4499: use of closed network connection Aug 7 23:51:17.854338279 (7h24m31.44139449s) [noise=true , address=104.36.82.35:4611 , piece_id=FE47XNPAFZAELTDVIO6C4URDMXRWEXCM37BFM4LANQQ6QDSU7HAQ, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zkqEMSKx7zSHPFNZ51b5TLzL2cFQL8VyxAG14Lg1pT8zAu3tc): protocol: write tcp 5.161.89.252:46266->104.36.82.35:4611: use of closed network connection; write tcp 5.161.89.252:46266->104.36.82.35:4611: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:46266->104.36.82.35:4611: use of closed network connection Aug 7 23:51:17.854399554 (7h24m31.441455765s) [noise=true , address=104.36.82.15:4288 , piece_id=KWW7E4ZMO42C3YEAH3WBCEYSW6TTNDX7TQ4MEITLSC2RR2CE4VDA, seg_pos={0 0}, upload=7600]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gPjPGuueZnizLT2QvJN9y7riwKiW5TWMkQtmGGPP7VZvkNoB7): protocol: read tcp 5.161.89.252:35428->104.36.82.15:4288: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:51:17.854419941 (7h24m31.441476162s) [seg_pos={0 0}, upload=7600]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:51:17.854472530 (7h24m31.441528751s) [upload=7600]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:51:17.854476738 (7h24m31.441532959s) [upload=7600]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:17.900817215 (7h24m31.487873465s) [upload=7600]: Done uploading segments... Aug 7 23:51:17.900834827 (7h24m31.487891079s) [upload=7600]: Upload finished. err: <nil> Aug 7 23:52:11.382772324 (7h25m24.969828534s) [upload=7616]: Starting upload Aug 7 23:52:11.382828459 (7h25m24.96988468s) [upload=7616]: Done starting upload Aug 7 23:52:11.382871529 (7h25m24.96992774s) [upload=7616]: Uploading segments... Aug 7 23:52:11.383053639 (7h25m24.97010986s) [upload=7616]: Got next segment. Inline: false Aug 7 23:52:11.383080680 (7h25m24.970136891s) [upload=7616]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:11.383084406 (7h25m24.970140617s) [upload=7616]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:11.371970391 (7h26m24.959026631s) [upload=7616]: Next segment err: context deadline exceeded Aug 7 23:53:11.372056972 (7h26m24.959113223s) [upload=7616]: Done uploading segments... Aug 7 23:53:11.372090856 (7h26m24.959147096s) [upload=7616]: Upload finished. err: context deadline exceeded Duration: 1m0.000851269s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)