Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
30.330116ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:20:45 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 30.351165ms
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration:
30.000836593s
FAIL (2025-08-07 21:20:33 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== r: <nil> Aug 7 21:12:30.034507380 (4h45m43.62156362s) [upload=4926]: Starting upload Aug 7 21:12:30.034584302 (4h45m43.621640544s) [upload=4926]: Done starting upload Aug 7 21:12:30.034643523 (4h45m43.621699754s) [upload=4926]: Uploading segments... Aug 7 21:12:30.034681965 (4h45m43.621738206s) [upload=4926]: Got next segment. Inline: true Aug 7 21:12:30.034687645 (4h45m43.621743886s) [upload=4926]: Waiting for error group managing segments... Aug 7 21:12:30.034730766 (4h45m43.621787007s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:30.034735976 (4h45m43.621792207s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:30.034739642 (4h45m43.621795883s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:00.022476261 (4h46m13.609532503s) [upload=4926]: Done uploading segments... Aug 7 21:13:00.022497211 (4h46m13.609553452s) [upload=4926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:15.702096720 (4h46m29.289152941s) [upload=4939]: Starting upload Aug 7 21:13:15.702154518 (4h46m29.289210739s) [upload=4939]: Done starting upload Aug 7 21:13:15.702196716 (4h46m29.289252928s) [upload=4939]: Uploading segments... Aug 7 21:13:15.702224368 (4h46m29.289280589s) [upload=4939]: Got next segment. Inline: true Aug 7 21:13:15.702228446 (4h46m29.289284657s) [upload=4939]: Waiting for error group managing segments... Aug 7 21:13:15.702261458 (4h46m29.289317669s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:13:15.702265306 (4h46m29.289321527s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:15.702267880 (4h46m29.289324101s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.689162551 (4h46m59.276218792s) [upload=4939]: Done uploading segments... Aug 7 21:13:45.689208046 (4h46m59.276264277s) [upload=4939]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:13.522007492 (4h47m27.109063713s) [upload=4955]: Starting upload Aug 7 21:14:13.522102209 (4h47m27.10915843s) [upload=4955]: Done starting upload Aug 7 21:14:13.522154938 (4h47m27.109211179s) [upload=4955]: Uploading segments... Aug 7 21:14:13.522235488 (4h47m27.109291729s) [upload=4955]: Got next segment. Inline: true Aug 7 21:14:13.522242682 (4h47m27.109298922s) [upload=4955]: Waiting for error group managing segments... Aug 7 21:14:13.522293467 (4h47m27.109349708s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:13.522299518 (4h47m27.109355759s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:13.522303275 (4h47m27.109359516s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:14:43.507883903 (4h47m57.094940144s) [upload=4955]: Done uploading segments... Aug 7 21:14:43.507905914 (4h47m57.094962156s) [upload=4955]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:54.825501522 (4h48m8.412557734s) [upload=4964]: Starting upload Aug 7 21:14:54.825580730 (4h48m8.412636951s) [upload=4964]: Done starting upload Aug 7 21:14:54.825629411 (4h48m8.412685632s) [upload=4964]: Uploading segments... Aug 7 21:14:54.825660400 (4h48m8.41271661s) [upload=4964]: Got next segment. Inline: true Aug 7 21:14:54.825664547 (4h48m8.412720758s) [upload=4964]: Waiting for error group managing segments... Aug 7 21:14:54.825705873 (4h48m8.412762115s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:54.825711855 (4h48m8.412768076s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:54.825715662 (4h48m8.412771883s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:15:24.812961328 (4h48m38.400017569s) [upload=4964]: Done uploading segments... Aug 7 21:15:24.812988770 (4h48m38.40004501s) [upload=4964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:40.993035983 (4h48m54.580092194s) [upload=4978]: Starting upload Aug 7 21:15:40.993105403 (4h48m54.580161614s) [upload=4978]: Done starting upload Aug 7 21:15:40.993154163 (4h48m54.580210375s) [upload=4978]: Uploading segments... Aug 7 21:15:40.993189519 (4h48m54.580245741s) [upload=4978]: Got next segment. Inline: true Aug 7 21:15:40.993194319 (4h48m54.58025054s) [upload=4978]: Waiting for error group managing segments... Aug 7 21:15:40.993226930 (4h48m54.580283151s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:40.993231167 (4h48m54.580287379s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:40.993234264 (4h48m54.580290474s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:10.982081529 (4h49m24.569137771s) [upload=4978]: Done uploading segments... Aug 7 21:16:10.982104473 (4h49m24.569160713s) [upload=4978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:20.227399180 (4h49m33.814455401s) [upload=4990]: Starting upload Aug 7 21:16:20.227450334 (4h49m33.814506545s) [upload=4990]: Done starting upload Aug 7 21:16:20.227491060 (4h49m33.814547272s) [upload=4990]: Uploading segments... Aug 7 21:16:20.227516609 (4h49m33.81457283s) [upload=4990]: Got next segment. Inline: true Aug 7 21:16:20.227520536 (4h49m33.814576757s) [upload=4990]: Waiting for error group managing segments... Aug 7 21:16:20.227552426 (4h49m33.814608636s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:20.227556263 (4h49m33.814612484s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:20.227558527 (4h49m33.814614739s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:50.216081431 (4h50m3.803137652s) [upload=4990]: Done uploading segments... Aug 7 21:16:50.216097492 (4h50m3.803153713s) [upload=4990]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:57.322533914 (4h50m10.909590156s) [upload=4995]: Starting upload Aug 7 21:16:57.322609355 (4h50m10.909665607s) [upload=4995]: Done starting upload Aug 7 21:16:57.322672883 (4h50m10.909729125s) [upload=4995]: Uploading segments... Aug 7 21:16:57.322711135 (4h50m10.909767376s) [upload=4995]: Got next segment. Inline: true Aug 7 21:16:57.322716685 (4h50m10.909772927s) [upload=4995]: Waiting for error group managing segments... Aug 7 21:16:57.322761279 (4h50m10.90981752s) [upload=4995]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:57.322767100 (4h50m10.909823341s) [upload=4995]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:57.322770697 (4h50m10.909826938s) [upload=4995]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:27.310589172 (4h50m40.897645383s) [upload=4995]: Done uploading segments... Aug 7 21:17:27.310603289 (4h50m40.897659499s) [upload=4995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:38.458031408 (4h50m52.045087639s) [upload=5010]: Starting upload Aug 7 21:17:38.458135221 (4h50m52.045191452s) [upload=5010]: Done starting upload Aug 7 21:17:38.458199552 (4h50m52.045255784s) [upload=5010]: Uploading segments... Aug 7 21:17:38.458275263 (4h50m52.045331504s) [upload=5010]: Got next segment. Inline: true Aug 7 21:17:38.458283519 (4h50m52.04533976s) [upload=5010]: Waiting for error group managing segments... Aug 7 21:17:38.458329695 (4h50m52.045385926s) [upload=5010]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:38.458335165 (4h50m52.045391407s) [upload=5010]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:38.458338662 (4h50m52.045394893s) [upload=5010]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:08.444937554 (4h51m22.031993765s) [upload=5010]: Done uploading segments... Aug 7 21:18:08.444950428 (4h51m22.032006639s) [upload=5010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:50.205377240 (4h52m3.792433451s) [upload=5028]: Starting upload Aug 7 21:18:50.205432072 (4h52m3.792488293s) [upload=5028]: Done starting upload Aug 7 21:18:50.205472938 (4h52m3.792529149s) [upload=5028]: Uploading segments... Aug 7 21:18:50.205500099 (4h52m3.792556311s) [upload=5028]: Got next segment. Inline: true Aug 7 21:18:50.205503937 (4h52m3.792560147s) [upload=5028]: Waiting for error group managing segments... Aug 7 21:18:50.205531809 (4h52m3.79258802s) [upload=5028]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:18:50.205535356 (4h52m3.792591577s) [upload=5028]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:50.205538692 (4h52m3.792594913s) [upload=5028]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:20.193730995 (4h52m33.780787246s) [upload=5028]: Done uploading segments... Aug 7 21:19:20.193750121 (4h52m33.780806372s) [upload=5028]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:03.998495850 (4h53m17.58555209s) [upload=5048]: Starting upload Aug 7 21:20:03.998559819 (4h53m17.585616059s) [upload=5048]: Done starting upload Aug 7 21:20:03.998621224 (4h53m17.585677465s) [upload=5048]: Uploading segments... Aug 7 21:20:03.998661639 (4h53m17.585717879s) [upload=5048]: Got next segment. Inline: true Aug 7 21:20:03.998667419 (4h53m17.58572365s) [upload=5048]: Waiting for error group managing segments... Aug 7 21:20:03.998710790 (4h53m17.585767032s) [upload=5048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:20:03.998716000 (4h53m17.585772242s) [upload=5048]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:03.998719587 (4h53m17.585775818s) [upload=5048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:20:33.984947930 (4h53m47.572004141s) [upload=5048]: Done uploading segments... Aug 7 21:20:33.985025244 (4h53m47.572081455s) [upload=5048]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.0008516s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration:
30.000881081s
FAIL (2025-08-07 21:20:31 UTC)
operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.000897541s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration:
30.001375574s
FAIL (2025-08-07 21:20:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001394399s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration:
1m0.001023189s
FAIL (2025-08-07 21:20:30 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2776712s) [upload=4897]: Done uploading segments... Aug 7 21:10:58.775736111 (4h44m12.362792352s) [upload=4897]: Upload finished. err: <nil> Aug 7 21:11:31.613155564 (4h44m45.200211785s) [upload=4908]: Starting upload Aug 7 21:11:31.613232477 (4h44m45.200288698s) [upload=4908]: Done starting upload Aug 7 21:11:31.613307799 (4h44m45.20036401s) [upload=4908]: Uploading segments... Aug 7 21:11:31.613347051 (4h44m45.200403272s) [upload=4908]: Got next segment. Inline: true Aug 7 21:11:31.613352932 (4h44m45.200409153s) [upload=4908]: Waiting for error group managing segments... Aug 7 21:11:31.613404179 (4h44m45.2004604s) [upload=4908]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:11:31.613410591 (4h44m45.200466842s) [upload=4908]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:31.613414508 (4h44m45.200470729s) [upload=4908]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.682826735 (4h44m45.269882977s) [upload=4908]: Done uploading segments... Aug 7 21:11:31.682853265 (4h44m45.269909476s) [upload=4908]: Upload finished. err: <nil> Aug 7 21:11:51.754258397 (4h45m5.341314618s) [upload=4915]: Starting upload Aug 7 21:11:51.754330331 (4h45m5.341386542s) [upload=4915]: Done starting upload Aug 7 21:11:51.754383972 (4h45m5.341440193s) [upload=4915]: Uploading segments... Aug 7 21:11:51.754417083 (4h45m5.341473294s) [upload=4915]: Got next segment. Inline: true Aug 7 21:11:51.754420630 (4h45m5.34147684s) [upload=4915]: Waiting for error group managing segments... Aug 7 21:11:51.754453131 (4h45m5.341509352s) [upload=4915]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:11:51.754457108 (4h45m5.341513329s) [upload=4915]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:51.754460725 (4h45m5.341516936s) [upload=4915]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:11:51.830658438 (4h45m5.417714649s) [upload=4915]: Done uploading segments... Aug 7 21:11:51.830673094 (4h45m5.417729315s) [upload=4915]: Upload finished. err: <nil> Aug 7 21:12:54.778494646 (4h46m8.365550856s) [upload=4932]: Starting upload Aug 7 21:12:54.778560719 (4h46m8.36561693s) [upload=4932]: Done starting upload Aug 7 21:12:54.778608638 (4h46m8.365664859s) [upload=4932]: Uploading segments... Aug 7 21:12:54.778637322 (4h46m8.365693543s) [upload=4932]: Got next segment. Inline: true Aug 7 21:12:54.778640999 (4h46m8.36569722s) [upload=4932]: Waiting for error group managing segments... Aug 7 21:12:54.778677527 (4h46m8.365733739s) [upload=4932]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:54.778681655 (4h46m8.365737876s) [upload=4932]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:54.778684771 (4h46m8.365740983s) [upload=4932]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:12:54.868239378 (4h46m8.455295599s) [upload=4932]: Done uploading segments... Aug 7 21:12:54.868253865 (4h46m8.455310086s) [upload=4932]: Upload finished. err: <nil> Aug 7 21:13:23.238906839 (4h46m36.82596306s) [upload=4942]: Starting upload Aug 7 21:13:23.238959356 (4h46m36.826015567s) [upload=4942]: Done starting upload Aug 7 21:13:23.239011013 (4h46m36.826067234s) [upload=4942]: Uploading segments... Aug 7 21:13:23.239039727 (4h46m36.826095937s) [upload=4942]: Got next segment. Inline: true Aug 7 21:13:23.239043483 (4h46m36.826099695s) [upload=4942]: Waiting for error group managing segments... Aug 7 21:13:23.239098526 (4h46m36.826154737s) [upload=4942]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:13:23.239104707 (4h46m36.826160919s) [upload=4942]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:23.239108474 (4h46m36.826164696s) [upload=4942]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:23.309357825 (4h46m36.896414127s) [upload=4942]: Done uploading segments... Aug 7 21:13:23.309383974 (4h46m36.896440216s) [upload=4942]: Upload finished. err: <nil> Aug 7 21:14:18.294119361 (4h47m31.881175572s) [upload=4957]: Starting upload Aug 7 21:14:18.294187468 (4h47m31.881243679s) [upload=4957]: Done starting upload Aug 7 21:14:18.294261516 (4h47m31.881317737s) [upload=4957]: Uploading segments... Aug 7 21:14:18.294305068 (4h47m31.881361289s) [upload=4957]: Got next segment. Inline: true Aug 7 21:14:18.294311409 (4h47m31.881367621s) [upload=4957]: Waiting for error group managing segments... Aug 7 21:14:18.294349491 (4h47m31.881405712s) [upload=4957]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:18.294354059 (4h47m31.88141028s) [upload=4957]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:18.294356643 (4h47m31.881412854s) [upload=4957]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:15:18.280461255 (4h48m31.867517496s) [upload=4957]: Done uploading segments... Aug 7 21:15:18.280484308 (4h48m31.867540559s) [upload=4957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:18.387536247 (4h48m31.974592488s) [upload=4974]: Starting upload Aug 7 21:15:18.387607770 (4h48m31.974664011s) [upload=4974]: Done starting upload Aug 7 21:15:18.387666109 (4h48m31.974722361s) [upload=4974]: Uploading segments... Aug 7 21:15:18.387706996 (4h48m31.974763236s) [upload=4974]: Got next segment. Inline: true Aug 7 21:15:18.387712766 (4h48m31.974769007s) [upload=4974]: Waiting for error group managing segments... Aug 7 21:15:18.387755796 (4h48m31.974812038s) [upload=4974]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:18.387761227 (4h48m31.974817468s) [upload=4974]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:18.387764853 (4h48m31.974821095s) [upload=4974]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:18.373970870 (4h49m31.961027121s) [upload=4974]: Done uploading segments... Aug 7 21:16:18.374035110 (4h49m31.961091352s) [upload=4974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:18.454541239 (4h49m32.04159748s) [upload=4989]: Starting upload Aug 7 21:16:18.454621288 (4h49m32.041677529s) [upload=4989]: Done starting upload Aug 7 21:16:18.454699754 (4h49m32.041756005s) [upload=4989]: Uploading segments... Aug 7 21:16:18.454758153 (4h49m32.041814395s) [upload=4989]: Got next segment. Inline: true Aug 7 21:16:18.454765147 (4h49m32.041821398s) [upload=4989]: Waiting for error group managing segments... Aug 7 21:16:18.454813777 (4h49m32.041870018s) [upload=4989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:18.454819818 (4h49m32.04187606s) [upload=4989]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:18.454823425 (4h49m32.041879666s) [upload=4989]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:18.435319201 (4h50m32.022375413s) [upload=4989]: Done uploading segments... Aug 7 21:17:18.435335041 (4h50m32.022391262s) [upload=4989]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:29.913551414 (4h50m43.500607635s) [upload=5005]: Starting upload Aug 7 21:17:29.913606246 (4h50m43.500662467s) [upload=5005]: Done starting upload Aug 7 21:17:29.913653174 (4h50m43.500709395s) [upload=5005]: Uploading segments... Aug 7 21:17:29.913680986 (4h50m43.500737207s) [upload=5005]: Got next segment. Inline: true Aug 7 21:17:29.913684613 (4h50m43.500740835s) [upload=5005]: Waiting for error group managing segments... Aug 7 21:17:29.913715249 (4h50m43.500771471s) [upload=5005]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:29.913719057 (4h50m43.500775278s) [upload=5005]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:29.913721331 (4h50m43.500777542s) [upload=5005]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:29.900370534 (4h51m43.487426745s) [upload=5005]: Done uploading segments... Aug 7 21:18:29.900390291 (4h51m43.487446542s) [upload=5005]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:29.988006379 (4h51m43.5750626s) [upload=5023]: Starting upload Aug 7 21:18:29.988053607 (4h51m43.575109818s) [upload=5023]: Done starting upload Aug 7 21:18:29.988109782 (4h51m43.575166003s) [upload=5023]: Uploading segments... Aug 7 21:18:29.988142012 (4h51m43.575198224s) [upload=5023]: Got next segment. Inline: true Aug 7 21:18:29.988147753 (4h51m43.575203973s) [upload=5023]: Waiting for error group managing segments... Aug 7 21:18:29.988197475 (4h51m43.575253687s) [upload=5023]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:18:29.988203116 (4h51m43.575259327s) [upload=5023]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:29.988206943 (4h51m43.575263164s) [upload=5023]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:29.967158594 (4h52m43.554214836s) [upload=5023]: Done uploading segments... Aug 7 21:19:29.967184642 (4h52m43.554240884s) [upload=5023]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:30.042411605 (4h52m43.629467826s) [upload=5040]: Starting upload Aug 7 21:19:30.042491373 (4h52m43.629547594s) [upload=5040]: Done starting upload Aug 7 21:19:30.042544984 (4h52m43.629601195s) [upload=5040]: Uploading segments... Aug 7 21:19:30.042585870 (4h52m43.629642091s) [upload=5040]: Got next segment. Inline: true Aug 7 21:19:30.042590809 (4h52m43.62964703s) [upload=5040]: Waiting for error group managing segments... Aug 7 21:19:30.042637226 (4h52m43.629693436s) [upload=5040]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:30.042642325 (4h52m43.629698546s) [upload=5040]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:30.042647144 (4h52m43.629703365s) [upload=5040]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:20:30.020861737 (4h53m43.607917958s) [upload=5040]: Done uploading segments... Aug 7 21:20:30.020878959 (4h53m43.60793518s) [upload=5040]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.001036865s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration:
1m0.001048481s
FAIL (2025-08-07 21:20:24 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
DJUTIWHY2LW3KBSLK76TKOVJNXC3RDV35KA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.288604064 (4h46m37.875660275s) [noise=true , address=104.36.82.23:4457 , piece_id=3LDUSHS23Z6ZBTDHVGVNUZRFA3H2XDC6B243ISKKCYJJG3PCRUBA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.289097404 (4h46m37.876153645s) [noise=true , address=104.36.82.44:4872 , piece_id=LM4MNB6MDHTN5MVJNGJOWX5Z5AKGICGK5JCLHBQBM5IR5PD3P2BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.303818073 (4h46m37.890874284s) [noise=true , address=104.36.82.20:4444 , piece_id=N3Z2NAQO5LHBPSABVMX4UBAAGUDDGDZCYGWX3NXWP6LUL5VJ74BA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.306609824 (4h46m37.893666035s) [noise=true , address=104.36.82.13:4190 , piece_id=PVB3Y7KXU47SOQCZBKW4AUZ56XMLMJQ23WPM73BKHUMZKARYOCCA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.307083188 (4h46m37.894139409s) [noise=true , address=104.36.82.46:4511 , piece_id=XTWLBKT2PX36GPDNTPPQ33HLJE2QMLNTVUMXVDEWJJO6IGC4HCXA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.333002740 (4h46m37.92005898s) [noise=true , address=104.36.82.11:4130 , piece_id=434EKDAN537AQFR342ZKLLENUIZAEK4PL2UTXDRJU7NJJ5HYHLUQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.335712287 (4h46m37.922768508s) [noise=true , address=79.127.220.97:4396 , piece_id=HD3NDYSYACLTEX2OYRUDYNK5J2DREH3DTNSJSDO745Q6NCC7AXGA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.364828819 (4h46m37.95188508s) [noise=true , address=104.36.82.54:4183 , piece_id=PIBOQ56366UKNWMH43GSS6TW66GFPMSJOB3UQ3TKMEYEYPKVVFOQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.415758456 (4h46m38.002814697s) [noise=true , address=104.36.82.37:4680 , piece_id=3LPKEUJ5HCZL4XY7YQOI2CROEHYNBFSZV6EUFDU2WV363FWQ644Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.427338251 (4h46m38.014394462s) [noise=true , address=104.36.82.43:4849 , piece_id=ZVVOBDRSIRLKPAOF23RMYRBXRNCNMQF3TJXYAW77ABOTGYUMXGHQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.490461549 (4h46m38.07751779s) [noise=true , address=79.127.201.215:4354 , piece_id=4VVIMMVTX6SCVPFAODIC7JS5PVULFS2SCXMKFJKA72NXKEK62K3Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.581523329 (4h46m38.16857954s) [noise=true , address=104.36.82.52:4108 , piece_id=AHQKVW323AG3TIIDCPX4HYFISVIY4V2476ZQM5GXLGLQD2PJ2KVA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.581552854 (4h46m38.168609065s) [seg_pos={0 0}, upload=4936]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:13:24.581774899 (4h46m38.16883112s) [noise=true , address=79.127.205.233:4643 , piece_id=IRDF6ZKJUCKCPSKAOCHW77H7O5ORLI43QULF2KWJ4O6SJTP373BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cVbdibJiX3T4fEUjeVBNNTTNaMb7m3QYDmxh17Dic5wVUQ9fx): protocol: expected piece hash; context canceled; EOF
Aug 7 21:13:24.581831154 (4h46m38.168887375s) [noise=true , address=104.36.82.62:4427 , piece_id=QMUVKOOD3THFKQAKBI3RQWH7JOP34MNNDOMKUXXBF4ZRVSYUJHJQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bx9fnLeXTPskzQB1Ad45qnq4CRivoGy7KXALY35dzVGF6S5Ep): protocol: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection
Aug 7 21:13:24.581866029 (4h46m38.168922239s) [noise=true , address=64.49.13.154:4933 , piece_id=HGHDC3TAI5HTPACRHKMJMLCMMF2AZOMT4DO5LIQEIECYYKCHMIRA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n5Mn8FNLXxFWtbbaMv8VuyxuXe6PiiMVtLETzm5aKfxhEykt3): protocol: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection
Aug 7 21:13:24.581904450 (4h46m38.168960661s) [noise=true , address=64.49.13.27:4963 , piece_id=MN6MBMOOPEIPZRW5R4NQM6AEE3AHYX33MIXOSIIZEGHDPJEKAOPA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq): protocol: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection
Aug 7 21:13:24.581925911 (4h46m38.168982132s) [seg_pos={0 0}, upload=4936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:13:24.582013845 (4h46m38.169070066s) [upload=4936]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:13:24.582018403 (4h46m38.169074624s) [upload=4936]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:15:00.743765287 (4h48m14.330821508s) [upload=4936]: Done uploading segments...
Aug 7 21:15:00.743795124 (4h48m14.330851345s) [upload=4936]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:15:00.812344761 (4h48m14.399400973s) [upload=4967]: Starting upload
Aug 7 21:15:00.812384405 (4h48m14.399440606s) [upload=4967]: Done starting upload
Aug 7 21:15:00.812509518 (4h48m14.399565729s) [upload=4967]: Uploading segments...
Aug 7 21:15:00.812592684 (4h48m14.399648895s) [upload=4967]: Got next segment. Inline: false
Aug 7 21:15:00.812634802 (4h48m14.399691023s) [upload=4967]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:15:00.812639241 (4h48m14.399695462s) [upload=4967]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:16:00.800009657 (4h49m14.387065878s) [upload=4967]: Next segment err: context deadline exceeded
Aug 7 21:16:00.800074909 (4h49m14.38713113s) [upload=4967]: Done uploading segments...
Aug 7 21:16:00.800088334 (4h49m14.387144545s) [upload=4967]: Upload finished. err: context deadline exceeded
Aug 7 21:16:09.662919397 (4h49m23.249975618s) [upload=4987]: Starting upload
Aug 7 21:16:09.663003895 (4h49m23.250060116s) [upload=4987]: Done starting upload
Aug 7 21:16:09.663197185 (4h49m23.250253426s) [upload=4987]: Uploading segments...
Aug 7 21:16:09.663304947 (4h49m23.250361248s) [upload=4987]: Got next segment. Inline: false
Aug 7 21:16:09.663375208 (4h49m23.250431459s) [upload=4987]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:16:09.663382832 (4h49m23.250439074s) [upload=4987]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:17:09.645661646 (4h50m23.232717887s) [upload=4987]: Next segment err: context deadline exceeded
Aug 7 21:17:09.645901063 (4h50m23.232957304s) [upload=4987]: Done uploading segments...
Aug 7 21:17:09.645976574 (4h50m23.233032815s) [upload=4987]: Upload finished. err: context deadline exceeded
Aug 7 21:17:10.545365898 (4h50m24.132422108s) [upload=5001]: Starting upload
Aug 7 21:17:10.545420008 (4h50m24.132476229s) [upload=5001]: Done starting upload
Aug 7 21:17:10.545476424 (4h50m24.132532645s) [upload=5001]: Uploading segments...
Aug 7 21:17:10.545636733 (4h50m24.132692944s) [upload=5001]: Got next segment. Inline: false
Aug 7 21:17:10.545661599 (4h50m24.13271782s) [upload=5001]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:17:10.545665476 (4h50m24.132721688s) [upload=5001]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:18:10.530620759 (4h51m24.11767701s) [upload=5001]: Next segment err: context deadline exceeded
Aug 7 21:18:10.530761893 (4h51m24.117818133s) [upload=5001]: Done uploading segments...
Aug 7 21:18:10.530782391 (4h51m24.117838632s) [upload=5001]: Upload finished. err: context deadline exceeded
Aug 7 21:18:10.615523121 (4h51m24.202579362s) [upload=5020]: Starting upload
Aug 7 21:18:10.615603712 (4h51m24.202659963s) [upload=5020]: Done starting upload
Aug 7 21:18:10.615743673 (4h51m24.202799914s) [upload=5020]: Uploading segments...
Aug 7 21:18:10.616023836 (4h51m24.203080087s) [upload=5020]: Got next segment. Inline: false
Aug 7 21:18:10.616082886 (4h51m24.203139138s) [upload=5020]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:18:10.616091172 (4h51m24.203147413s) [upload=5020]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:19:10.603574083 (4h52m24.190630324s) [upload=5020]: Next segment err: context deadline exceeded
Aug 7 21:19:10.603816245 (4h52m24.190872486s) [upload=5020]: Done uploading segments...
Aug 7 21:19:10.603869965 (4h52m24.190926207s) [upload=5020]: Upload finished. err: context deadline exceeded
Aug 7 21:19:24.417770609 (4h52m38.004826851s) [upload=5037]: Starting upload
Aug 7 21:19:24.417854846 (4h52m38.004911088s) [upload=5037]: Done starting upload
Aug 7 21:19:24.417989017 (4h52m38.005045258s) [upload=5037]: Uploading segments...
Aug 7 21:19:24.418253952 (4h52m38.005310193s) [upload=5037]: Got next segment. Inline: false
Aug 7 21:19:24.418302842 (4h52m38.005359084s) [upload=5037]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:19:24.418308183 (4h52m38.005364424s) [upload=5037]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:20:24.399165083 (4h53m37.986221304s) [upload=5037]: Next segment err: context deadline exceeded
Aug 7 21:20:24.399217411 (4h53m37.986273622s) [upload=5037]: Done uploading segments...
Aug 7 21:20:24.399231628 (4h53m37.986287849s) [upload=5037]: Upload finished. err: context deadline exceeded
Duration: 1m0.001067757s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
11 out of 60 is failed (threshold=3) [...............................................XX..XXXXXXXXX] duration:
30.000789441s
FAIL (2025-08-07 21:20:08 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000812834s
core
uplink
us1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
39.725283ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:20:04 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 39.745531ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration:
30.00103077s
FAIL (2025-08-07 21:20:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001049625s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [....................................................XX.XXXXX] duration:
1m0.000747616s
FAIL (2025-08-07 21:19:51 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000758566s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
1.262049311s
FAIL (2025-08-07 21:19:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment
Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments...
Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF
Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection
Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection
Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments...
Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil>
Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload
Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload
Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments...
Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false
Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests
Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments...
Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.262068497s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration:
30.000570725s
FAIL (2025-08-07 21:19:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000589399s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
26.229712ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:19:37 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 26.252615ms
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [................................................XX..XXXXXXXX] duration:
30.000918728s
FAIL (2025-08-07 21:19:33 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000937432s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration:
30.000510055s
FAIL (2025-08-07 21:19:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000524151s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration:
1m0.001346544s
FAIL (2025-08-07 21:19:29 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:10:26.611321150 (4h43m40.198377371s) [upload=4887]: Done uploading segments... Aug 7 21:10:26.611339073 (4h43m40.198395294s) [upload=4887]: Upload finished. err: <nil> Aug 7 21:10:58.699165641 (4h44m12.286221892s) [upload=4897]: Starting upload Aug 7 21:10:58.699245781 (4h44m12.286302022s) [upload=4897]: Done starting upload Aug 7 21:10:58.699314669 (4h44m12.286370911s) [upload=4897]: Uploading segments... Aug 7 21:10:58.699373410 (4h44m12.286429651s) [upload=4897]: Got next segment. Inline: true Aug 7 21:10:58.699380724 (4h44m12.286436964s) [upload=4897]: Waiting for error group managing segments... Aug 7 21:10:58.699426338 (4h44m12.286482579s) [upload=4897]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:10:58.699432380 (4h44m12.28648862s) [upload=4897]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:58.699436066 (4h44m12.286492317s) [upload=4897]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:10:58.775720472 (4h44m12.362776712s) [upload=4897]: Done uploading segments... Aug 7 21:10:58.775736111 (4h44m12.362792352s) [upload=4897]: Upload finished. err: <nil> Aug 7 21:11:31.613155564 (4h44m45.200211785s) [upload=4908]: Starting upload Aug 7 21:11:31.613232477 (4h44m45.200288698s) [upload=4908]: Done starting upload Aug 7 21:11:31.613307799 (4h44m45.20036401s) [upload=4908]: Uploading segments... Aug 7 21:11:31.613347051 (4h44m45.200403272s) [upload=4908]: Got next segment. Inline: true Aug 7 21:11:31.613352932 (4h44m45.200409153s) [upload=4908]: Waiting for error group managing segments... Aug 7 21:11:31.613404179 (4h44m45.2004604s) [upload=4908]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:11:31.613410591 (4h44m45.200466842s) [upload=4908]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:31.613414508 (4h44m45.200470729s) [upload=4908]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.682826735 (4h44m45.269882977s) [upload=4908]: Done uploading segments... Aug 7 21:11:31.682853265 (4h44m45.269909476s) [upload=4908]: Upload finished. err: <nil> Aug 7 21:11:51.754258397 (4h45m5.341314618s) [upload=4915]: Starting upload Aug 7 21:11:51.754330331 (4h45m5.341386542s) [upload=4915]: Done starting upload Aug 7 21:11:51.754383972 (4h45m5.341440193s) [upload=4915]: Uploading segments... Aug 7 21:11:51.754417083 (4h45m5.341473294s) [upload=4915]: Got next segment. Inline: true Aug 7 21:11:51.754420630 (4h45m5.34147684s) [upload=4915]: Waiting for error group managing segments... Aug 7 21:11:51.754453131 (4h45m5.341509352s) [upload=4915]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:11:51.754457108 (4h45m5.341513329s) [upload=4915]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:51.754460725 (4h45m5.341516936s) [upload=4915]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:11:51.830658438 (4h45m5.417714649s) [upload=4915]: Done uploading segments... Aug 7 21:11:51.830673094 (4h45m5.417729315s) [upload=4915]: Upload finished. err: <nil> Aug 7 21:12:54.778494646 (4h46m8.365550856s) [upload=4932]: Starting upload Aug 7 21:12:54.778560719 (4h46m8.36561693s) [upload=4932]: Done starting upload Aug 7 21:12:54.778608638 (4h46m8.365664859s) [upload=4932]: Uploading segments... Aug 7 21:12:54.778637322 (4h46m8.365693543s) [upload=4932]: Got next segment. Inline: true Aug 7 21:12:54.778640999 (4h46m8.36569722s) [upload=4932]: Waiting for error group managing segments... Aug 7 21:12:54.778677527 (4h46m8.365733739s) [upload=4932]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:54.778681655 (4h46m8.365737876s) [upload=4932]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:54.778684771 (4h46m8.365740983s) [upload=4932]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:12:54.868239378 (4h46m8.455295599s) [upload=4932]: Done uploading segments... Aug 7 21:12:54.868253865 (4h46m8.455310086s) [upload=4932]: Upload finished. err: <nil> Aug 7 21:13:23.238906839 (4h46m36.82596306s) [upload=4942]: Starting upload Aug 7 21:13:23.238959356 (4h46m36.826015567s) [upload=4942]: Done starting upload Aug 7 21:13:23.239011013 (4h46m36.826067234s) [upload=4942]: Uploading segments... Aug 7 21:13:23.239039727 (4h46m36.826095937s) [upload=4942]: Got next segment. Inline: true Aug 7 21:13:23.239043483 (4h46m36.826099695s) [upload=4942]: Waiting for error group managing segments... Aug 7 21:13:23.239098526 (4h46m36.826154737s) [upload=4942]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:13:23.239104707 (4h46m36.826160919s) [upload=4942]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:23.239108474 (4h46m36.826164696s) [upload=4942]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:23.309357825 (4h46m36.896414127s) [upload=4942]: Done uploading segments... Aug 7 21:13:23.309383974 (4h46m36.896440216s) [upload=4942]: Upload finished. err: <nil> Aug 7 21:14:18.294119361 (4h47m31.881175572s) [upload=4957]: Starting upload Aug 7 21:14:18.294187468 (4h47m31.881243679s) [upload=4957]: Done starting upload Aug 7 21:14:18.294261516 (4h47m31.881317737s) [upload=4957]: Uploading segments... Aug 7 21:14:18.294305068 (4h47m31.881361289s) [upload=4957]: Got next segment. Inline: true Aug 7 21:14:18.294311409 (4h47m31.881367621s) [upload=4957]: Waiting for error group managing segments... Aug 7 21:14:18.294349491 (4h47m31.881405712s) [upload=4957]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:18.294354059 (4h47m31.88141028s) [upload=4957]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:18.294356643 (4h47m31.881412854s) [upload=4957]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:15:18.280461255 (4h48m31.867517496s) [upload=4957]: Done uploading segments... Aug 7 21:15:18.280484308 (4h48m31.867540559s) [upload=4957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:18.387536247 (4h48m31.974592488s) [upload=4974]: Starting upload Aug 7 21:15:18.387607770 (4h48m31.974664011s) [upload=4974]: Done starting upload Aug 7 21:15:18.387666109 (4h48m31.974722361s) [upload=4974]: Uploading segments... Aug 7 21:15:18.387706996 (4h48m31.974763236s) [upload=4974]: Got next segment. Inline: true Aug 7 21:15:18.387712766 (4h48m31.974769007s) [upload=4974]: Waiting for error group managing segments... Aug 7 21:15:18.387755796 (4h48m31.974812038s) [upload=4974]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:18.387761227 (4h48m31.974817468s) [upload=4974]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:18.387764853 (4h48m31.974821095s) [upload=4974]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:18.373970870 (4h49m31.961027121s) [upload=4974]: Done uploading segments... Aug 7 21:16:18.374035110 (4h49m31.961091352s) [upload=4974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:18.454541239 (4h49m32.04159748s) [upload=4989]: Starting upload Aug 7 21:16:18.454621288 (4h49m32.041677529s) [upload=4989]: Done starting upload Aug 7 21:16:18.454699754 (4h49m32.041756005s) [upload=4989]: Uploading segments... Aug 7 21:16:18.454758153 (4h49m32.041814395s) [upload=4989]: Got next segment. Inline: true Aug 7 21:16:18.454765147 (4h49m32.041821398s) [upload=4989]: Waiting for error group managing segments... Aug 7 21:16:18.454813777 (4h49m32.041870018s) [upload=4989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:18.454819818 (4h49m32.04187606s) [upload=4989]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:18.454823425 (4h49m32.041879666s) [upload=4989]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:18.435319201 (4h50m32.022375413s) [upload=4989]: Done uploading segments... Aug 7 21:17:18.435335041 (4h50m32.022391262s) [upload=4989]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:29.913551414 (4h50m43.500607635s) [upload=5005]: Starting upload Aug 7 21:17:29.913606246 (4h50m43.500662467s) [upload=5005]: Done starting upload Aug 7 21:17:29.913653174 (4h50m43.500709395s) [upload=5005]: Uploading segments... Aug 7 21:17:29.913680986 (4h50m43.500737207s) [upload=5005]: Got next segment. Inline: true Aug 7 21:17:29.913684613 (4h50m43.500740835s) [upload=5005]: Waiting for error group managing segments... Aug 7 21:17:29.913715249 (4h50m43.500771471s) [upload=5005]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:29.913719057 (4h50m43.500775278s) [upload=5005]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:29.913721331 (4h50m43.500777542s) [upload=5005]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:29.900370534 (4h51m43.487426745s) [upload=5005]: Done uploading segments... Aug 7 21:18:29.900390291 (4h51m43.487446542s) [upload=5005]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:29.988006379 (4h51m43.5750626s) [upload=5023]: Starting upload Aug 7 21:18:29.988053607 (4h51m43.575109818s) [upload=5023]: Done starting upload Aug 7 21:18:29.988109782 (4h51m43.575166003s) [upload=5023]: Uploading segments... Aug 7 21:18:29.988142012 (4h51m43.575198224s) [upload=5023]: Got next segment. Inline: true Aug 7 21:18:29.988147753 (4h51m43.575203973s) [upload=5023]: Waiting for error group managing segments... Aug 7 21:18:29.988197475 (4h51m43.575253687s) [upload=5023]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:18:29.988203116 (4h51m43.575259327s) [upload=5023]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:29.988206943 (4h51m43.575263164s) [upload=5023]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:29.967158594 (4h52m43.554214836s) [upload=5023]: Done uploading segments... Aug 7 21:19:29.967184642 (4h52m43.554240884s) [upload=5023]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.00135987s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration:
30.000428614s
FAIL (2025-08-07 21:19:20 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad=4903]: Upload finished. err: <nil> Aug 7 21:12:08.543001654 (4h45m22.130057896s) [upload=4919]: Starting upload Aug 7 21:12:08.543123191 (4h45m22.130179432s) [upload=4919]: Done starting upload Aug 7 21:12:08.543216246 (4h45m22.130272487s) [upload=4919]: Uploading segments... Aug 7 21:12:08.543261810 (4h45m22.130318051s) [upload=4919]: Got next segment. Inline: true Aug 7 21:12:08.543267651 (4h45m22.130323892s) [upload=4919]: Waiting for error group managing segments... Aug 7 21:12:08.543313246 (4h45m22.130369488s) [upload=4919]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:08.543319278 (4h45m22.130375519s) [upload=4919]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:08.543323075 (4h45m22.130379316s) [upload=4919]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:12:08.630954371 (4h45m22.218010612s) [upload=4919]: Done uploading segments... Aug 7 21:12:08.630979197 (4h45m22.218035449s) [upload=4919]: Upload finished. err: <nil> Aug 7 21:12:30.034507380 (4h45m43.62156362s) [upload=4926]: Starting upload Aug 7 21:12:30.034584302 (4h45m43.621640544s) [upload=4926]: Done starting upload Aug 7 21:12:30.034643523 (4h45m43.621699754s) [upload=4926]: Uploading segments... Aug 7 21:12:30.034681965 (4h45m43.621738206s) [upload=4926]: Got next segment. Inline: true Aug 7 21:12:30.034687645 (4h45m43.621743886s) [upload=4926]: Waiting for error group managing segments... Aug 7 21:12:30.034730766 (4h45m43.621787007s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:30.034735976 (4h45m43.621792207s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:30.034739642 (4h45m43.621795883s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:00.022476261 (4h46m13.609532503s) [upload=4926]: Done uploading segments... Aug 7 21:13:00.022497211 (4h46m13.609553452s) [upload=4926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:15.702096720 (4h46m29.289152941s) [upload=4939]: Starting upload Aug 7 21:13:15.702154518 (4h46m29.289210739s) [upload=4939]: Done starting upload Aug 7 21:13:15.702196716 (4h46m29.289252928s) [upload=4939]: Uploading segments... Aug 7 21:13:15.702224368 (4h46m29.289280589s) [upload=4939]: Got next segment. Inline: true Aug 7 21:13:15.702228446 (4h46m29.289284657s) [upload=4939]: Waiting for error group managing segments... Aug 7 21:13:15.702261458 (4h46m29.289317669s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:13:15.702265306 (4h46m29.289321527s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:15.702267880 (4h46m29.289324101s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.689162551 (4h46m59.276218792s) [upload=4939]: Done uploading segments... Aug 7 21:13:45.689208046 (4h46m59.276264277s) [upload=4939]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:13.522007492 (4h47m27.109063713s) [upload=4955]: Starting upload Aug 7 21:14:13.522102209 (4h47m27.10915843s) [upload=4955]: Done starting upload Aug 7 21:14:13.522154938 (4h47m27.109211179s) [upload=4955]: Uploading segments... Aug 7 21:14:13.522235488 (4h47m27.109291729s) [upload=4955]: Got next segment. Inline: true Aug 7 21:14:13.522242682 (4h47m27.109298922s) [upload=4955]: Waiting for error group managing segments... Aug 7 21:14:13.522293467 (4h47m27.109349708s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:13.522299518 (4h47m27.109355759s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:13.522303275 (4h47m27.109359516s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:14:43.507883903 (4h47m57.094940144s) [upload=4955]: Done uploading segments... Aug 7 21:14:43.507905914 (4h47m57.094962156s) [upload=4955]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:54.825501522 (4h48m8.412557734s) [upload=4964]: Starting upload Aug 7 21:14:54.825580730 (4h48m8.412636951s) [upload=4964]: Done starting upload Aug 7 21:14:54.825629411 (4h48m8.412685632s) [upload=4964]: Uploading segments... Aug 7 21:14:54.825660400 (4h48m8.41271661s) [upload=4964]: Got next segment. Inline: true Aug 7 21:14:54.825664547 (4h48m8.412720758s) [upload=4964]: Waiting for error group managing segments... Aug 7 21:14:54.825705873 (4h48m8.412762115s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:54.825711855 (4h48m8.412768076s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:54.825715662 (4h48m8.412771883s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:15:24.812961328 (4h48m38.400017569s) [upload=4964]: Done uploading segments... Aug 7 21:15:24.812988770 (4h48m38.40004501s) [upload=4964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:40.993035983 (4h48m54.580092194s) [upload=4978]: Starting upload Aug 7 21:15:40.993105403 (4h48m54.580161614s) [upload=4978]: Done starting upload Aug 7 21:15:40.993154163 (4h48m54.580210375s) [upload=4978]: Uploading segments... Aug 7 21:15:40.993189519 (4h48m54.580245741s) [upload=4978]: Got next segment. Inline: true Aug 7 21:15:40.993194319 (4h48m54.58025054s) [upload=4978]: Waiting for error group managing segments... Aug 7 21:15:40.993226930 (4h48m54.580283151s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:40.993231167 (4h48m54.580287379s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:40.993234264 (4h48m54.580290474s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:10.982081529 (4h49m24.569137771s) [upload=4978]: Done uploading segments... Aug 7 21:16:10.982104473 (4h49m24.569160713s) [upload=4978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:20.227399180 (4h49m33.814455401s) [upload=4990]: Starting upload Aug 7 21:16:20.227450334 (4h49m33.814506545s) [upload=4990]: Done starting upload Aug 7 21:16:20.227491060 (4h49m33.814547272s) [upload=4990]: Uploading segments... Aug 7 21:16:20.227516609 (4h49m33.81457283s) [upload=4990]: Got next segment. Inline: true Aug 7 21:16:20.227520536 (4h49m33.814576757s) [upload=4990]: Waiting for error group managing segments... Aug 7 21:16:20.227552426 (4h49m33.814608636s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:20.227556263 (4h49m33.814612484s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:20.227558527 (4h49m33.814614739s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:50.216081431 (4h50m3.803137652s) [upload=4990]: Done uploading segments... Aug 7 21:16:50.216097492 (4h50m3.803153713s) [upload=4990]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:57.322533914 (4h50m10.909590156s) [upload=4995]: Starting upload Aug 7 21:16:57.322609355 (4h50m10.909665607s) [upload=4995]: Done starting upload Aug 7 21:16:57.322672883 (4h50m10.909729125s) [upload=4995]: Uploading segments... Aug 7 21:16:57.322711135 (4h50m10.909767376s) [upload=4995]: Got next segment. Inline: true Aug 7 21:16:57.322716685 (4h50m10.909772927s) [upload=4995]: Waiting for error group managing segments... Aug 7 21:16:57.322761279 (4h50m10.90981752s) [upload=4995]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:57.322767100 (4h50m10.909823341s) [upload=4995]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:57.322770697 (4h50m10.909826938s) [upload=4995]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:27.310589172 (4h50m40.897645383s) [upload=4995]: Done uploading segments... Aug 7 21:17:27.310603289 (4h50m40.897659499s) [upload=4995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:38.458031408 (4h50m52.045087639s) [upload=5010]: Starting upload Aug 7 21:17:38.458135221 (4h50m52.045191452s) [upload=5010]: Done starting upload Aug 7 21:17:38.458199552 (4h50m52.045255784s) [upload=5010]: Uploading segments... Aug 7 21:17:38.458275263 (4h50m52.045331504s) [upload=5010]: Got next segment. Inline: true Aug 7 21:17:38.458283519 (4h50m52.04533976s) [upload=5010]: Waiting for error group managing segments... Aug 7 21:17:38.458329695 (4h50m52.045385926s) [upload=5010]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:38.458335165 (4h50m52.045391407s) [upload=5010]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:38.458338662 (4h50m52.045394893s) [upload=5010]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:08.444937554 (4h51m22.031993765s) [upload=5010]: Done uploading segments... Aug 7 21:18:08.444950428 (4h51m22.032006639s) [upload=5010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:50.205377240 (4h52m3.792433451s) [upload=5028]: Starting upload Aug 7 21:18:50.205432072 (4h52m3.792488293s) [upload=5028]: Done starting upload Aug 7 21:18:50.205472938 (4h52m3.792529149s) [upload=5028]: Uploading segments... Aug 7 21:18:50.205500099 (4h52m3.792556311s) [upload=5028]: Got next segment. Inline: true Aug 7 21:18:50.205503937 (4h52m3.792560147s) [upload=5028]: Waiting for error group managing segments... Aug 7 21:18:50.205531809 (4h52m3.79258802s) [upload=5028]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:18:50.205535356 (4h52m3.792591577s) [upload=5028]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:50.205538692 (4h52m3.792594913s) [upload=5028]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:20.193730995 (4h52m33.780787246s) [upload=5028]: Done uploading segments... Aug 7 21:19:20.193750121 (4h52m33.780806372s) [upload=5028]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.000449283s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
30.000739787s
FAIL (2025-08-07 21:19:18 UTC)
operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.000769422s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration:
1m0.001136648s
FAIL (2025-08-07 21:19:10 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
ess=104.36.82.54:4183 , piece_id=PIBOQ56366UKNWMH43GSS6TW66GFPMSJOB3UQ3TKMEYEYPKVVFOQ, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.169710120 (4h46m37.756766341s) [noise=true , address=104.36.82.52:4108 , piece_id=AHQKVW323AG3TIIDCPX4HYFISVIY4V2476ZQM5GXLGLQD2PJ2KVA, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.260747004 (4h46m37.847803255s) [upload=4936]: Next returned nil segment
Aug 7 21:13:24.260784884 (4h46m37.847841125s) [upload=4936]: Waiting for error group managing segments...
Aug 7 21:13:24.282697478 (4h46m37.869753699s) [noise=true , address=104.36.82.36:4652 , piece_id=4S6GW6S5QMKXREUOZ4K7S77A5FIA5OGIS6MIPH3X5WFAIUYUTKBQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.286011234 (4h46m37.873067475s) [noise=true , address=45.91.116.10:4113 , piece_id=NJYFA7PI7F62J2CSEDJUTIWHY2LW3KBSLK76TKOVJNXC3RDV35KA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.288604064 (4h46m37.875660275s) [noise=true , address=104.36.82.23:4457 , piece_id=3LDUSHS23Z6ZBTDHVGVNUZRFA3H2XDC6B243ISKKCYJJG3PCRUBA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.289097404 (4h46m37.876153645s) [noise=true , address=104.36.82.44:4872 , piece_id=LM4MNB6MDHTN5MVJNGJOWX5Z5AKGICGK5JCLHBQBM5IR5PD3P2BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.303818073 (4h46m37.890874284s) [noise=true , address=104.36.82.20:4444 , piece_id=N3Z2NAQO5LHBPSABVMX4UBAAGUDDGDZCYGWX3NXWP6LUL5VJ74BA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.306609824 (4h46m37.893666035s) [noise=true , address=104.36.82.13:4190 , piece_id=PVB3Y7KXU47SOQCZBKW4AUZ56XMLMJQ23WPM73BKHUMZKARYOCCA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.307083188 (4h46m37.894139409s) [noise=true , address=104.36.82.46:4511 , piece_id=XTWLBKT2PX36GPDNTPPQ33HLJE2QMLNTVUMXVDEWJJO6IGC4HCXA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.333002740 (4h46m37.92005898s) [noise=true , address=104.36.82.11:4130 , piece_id=434EKDAN537AQFR342ZKLLENUIZAEK4PL2UTXDRJU7NJJ5HYHLUQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.335712287 (4h46m37.922768508s) [noise=true , address=79.127.220.97:4396 , piece_id=HD3NDYSYACLTEX2OYRUDYNK5J2DREH3DTNSJSDO745Q6NCC7AXGA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.364828819 (4h46m37.95188508s) [noise=true , address=104.36.82.54:4183 , piece_id=PIBOQ56366UKNWMH43GSS6TW66GFPMSJOB3UQ3TKMEYEYPKVVFOQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.415758456 (4h46m38.002814697s) [noise=true , address=104.36.82.37:4680 , piece_id=3LPKEUJ5HCZL4XY7YQOI2CROEHYNBFSZV6EUFDU2WV363FWQ644Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.427338251 (4h46m38.014394462s) [noise=true , address=104.36.82.43:4849 , piece_id=ZVVOBDRSIRLKPAOF23RMYRBXRNCNMQF3TJXYAW77ABOTGYUMXGHQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.490461549 (4h46m38.07751779s) [noise=true , address=79.127.201.215:4354 , piece_id=4VVIMMVTX6SCVPFAODIC7JS5PVULFS2SCXMKFJKA72NXKEK62K3Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.581523329 (4h46m38.16857954s) [noise=true , address=104.36.82.52:4108 , piece_id=AHQKVW323AG3TIIDCPX4HYFISVIY4V2476ZQM5GXLGLQD2PJ2KVA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.581552854 (4h46m38.168609065s) [seg_pos={0 0}, upload=4936]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:13:24.581774899 (4h46m38.16883112s) [noise=true , address=79.127.205.233:4643 , piece_id=IRDF6ZKJUCKCPSKAOCHW77H7O5ORLI43QULF2KWJ4O6SJTP373BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cVbdibJiX3T4fEUjeVBNNTTNaMb7m3QYDmxh17Dic5wVUQ9fx): protocol: expected piece hash; context canceled; EOF
Aug 7 21:13:24.581831154 (4h46m38.168887375s) [noise=true , address=104.36.82.62:4427 , piece_id=QMUVKOOD3THFKQAKBI3RQWH7JOP34MNNDOMKUXXBF4ZRVSYUJHJQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bx9fnLeXTPskzQB1Ad45qnq4CRivoGy7KXALY35dzVGF6S5Ep): protocol: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection
Aug 7 21:13:24.581866029 (4h46m38.168922239s) [noise=true , address=64.49.13.154:4933 , piece_id=HGHDC3TAI5HTPACRHKMJMLCMMF2AZOMT4DO5LIQEIECYYKCHMIRA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n5Mn8FNLXxFWtbbaMv8VuyxuXe6PiiMVtLETzm5aKfxhEykt3): protocol: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection
Aug 7 21:13:24.581904450 (4h46m38.168960661s) [noise=true , address=64.49.13.27:4963 , piece_id=MN6MBMOOPEIPZRW5R4NQM6AEE3AHYX33MIXOSIIZEGHDPJEKAOPA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq): protocol: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection
Aug 7 21:13:24.581925911 (4h46m38.168982132s) [seg_pos={0 0}, upload=4936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:13:24.582013845 (4h46m38.169070066s) [upload=4936]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:13:24.582018403 (4h46m38.169074624s) [upload=4936]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:15:00.743765287 (4h48m14.330821508s) [upload=4936]: Done uploading segments...
Aug 7 21:15:00.743795124 (4h48m14.330851345s) [upload=4936]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:15:00.812344761 (4h48m14.399400973s) [upload=4967]: Starting upload
Aug 7 21:15:00.812384405 (4h48m14.399440606s) [upload=4967]: Done starting upload
Aug 7 21:15:00.812509518 (4h48m14.399565729s) [upload=4967]: Uploading segments...
Aug 7 21:15:00.812592684 (4h48m14.399648895s) [upload=4967]: Got next segment. Inline: false
Aug 7 21:15:00.812634802 (4h48m14.399691023s) [upload=4967]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:15:00.812639241 (4h48m14.399695462s) [upload=4967]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:16:00.800009657 (4h49m14.387065878s) [upload=4967]: Next segment err: context deadline exceeded
Aug 7 21:16:00.800074909 (4h49m14.38713113s) [upload=4967]: Done uploading segments...
Aug 7 21:16:00.800088334 (4h49m14.387144545s) [upload=4967]: Upload finished. err: context deadline exceeded
Aug 7 21:16:09.662919397 (4h49m23.249975618s) [upload=4987]: Starting upload
Aug 7 21:16:09.663003895 (4h49m23.250060116s) [upload=4987]: Done starting upload
Aug 7 21:16:09.663197185 (4h49m23.250253426s) [upload=4987]: Uploading segments...
Aug 7 21:16:09.663304947 (4h49m23.250361248s) [upload=4987]: Got next segment. Inline: false
Aug 7 21:16:09.663375208 (4h49m23.250431459s) [upload=4987]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:16:09.663382832 (4h49m23.250439074s) [upload=4987]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:17:09.645661646 (4h50m23.232717887s) [upload=4987]: Next segment err: context deadline exceeded
Aug 7 21:17:09.645901063 (4h50m23.232957304s) [upload=4987]: Done uploading segments...
Aug 7 21:17:09.645976574 (4h50m23.233032815s) [upload=4987]: Upload finished. err: context deadline exceeded
Aug 7 21:17:10.545365898 (4h50m24.132422108s) [upload=5001]: Starting upload
Aug 7 21:17:10.545420008 (4h50m24.132476229s) [upload=5001]: Done starting upload
Aug 7 21:17:10.545476424 (4h50m24.132532645s) [upload=5001]: Uploading segments...
Aug 7 21:17:10.545636733 (4h50m24.132692944s) [upload=5001]: Got next segment. Inline: false
Aug 7 21:17:10.545661599 (4h50m24.13271782s) [upload=5001]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:17:10.545665476 (4h50m24.132721688s) [upload=5001]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:18:10.530620759 (4h51m24.11767701s) [upload=5001]: Next segment err: context deadline exceeded
Aug 7 21:18:10.530761893 (4h51m24.117818133s) [upload=5001]: Done uploading segments...
Aug 7 21:18:10.530782391 (4h51m24.117838632s) [upload=5001]: Upload finished. err: context deadline exceeded
Aug 7 21:18:10.615523121 (4h51m24.202579362s) [upload=5020]: Starting upload
Aug 7 21:18:10.615603712 (4h51m24.202659963s) [upload=5020]: Done starting upload
Aug 7 21:18:10.615743673 (4h51m24.202799914s) [upload=5020]: Uploading segments...
Aug 7 21:18:10.616023836 (4h51m24.203080087s) [upload=5020]: Got next segment. Inline: false
Aug 7 21:18:10.616082886 (4h51m24.203139138s) [upload=5020]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:18:10.616091172 (4h51m24.203147413s) [upload=5020]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:19:10.603574083 (4h52m24.190630324s) [upload=5020]: Next segment err: context deadline exceeded
Aug 7 21:19:10.603816245 (4h52m24.190872486s) [upload=5020]: Done uploading segments...
Aug 7 21:19:10.603869965 (4h52m24.190926207s) [upload=5020]: Upload finished. err: context deadline exceeded
Duration: 1m0.001153199s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
1.036362s
FAIL (2025-08-07 21:19:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment
Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments...
Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF
Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection
Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection
Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments...
Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil>
Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload
Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload
Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments...
Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false
Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests
Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments...
Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.036391235s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration:
30.000727199s
FAIL (2025-08-07 21:18:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000954955s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.....................................................XX.XXXX] duration:
1m0.00124501s
FAIL (2025-08-07 21:18:51 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001258795s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
30.000913301s
FAIL (2025-08-07 21:18:37 UTC)
operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.000931004s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration:
30.001050479s
FAIL (2025-08-07 21:18:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001065267s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
1m0.000982407s
FAIL (2025-08-07 21:18:29 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== RequestItem_ObjectCommit Aug 7 21:09:18.367751803 (4h42m31.954808044s) [upload=4869]: Done uploading segments... Aug 7 21:09:18.367774827 (4h42m31.954831078s) [upload=4869]: Upload finished. err: <nil> Aug 7 21:10:26.543295840 (4h43m40.130352072s) [upload=4887]: Starting upload Aug 7 21:10:26.543341496 (4h43m40.130397717s) [upload=4887]: Done starting upload Aug 7 21:10:26.543385698 (4h43m40.130441919s) [upload=4887]: Uploading segments... Aug 7 21:10:26.543414672 (4h43m40.130470883s) [upload=4887]: Got next segment. Inline: true Aug 7 21:10:26.543418339 (4h43m40.13047455s) [upload=4887]: Waiting for error group managing segments... Aug 7 21:10:26.543446882 (4h43m40.130503103s) [upload=4887]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:10:26.543450790 (4h43m40.130507011s) [upload=4887]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:26.543453054 (4h43m40.130509265s) [upload=4887]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:10:26.611321150 (4h43m40.198377371s) [upload=4887]: Done uploading segments... Aug 7 21:10:26.611339073 (4h43m40.198395294s) [upload=4887]: Upload finished. err: <nil> Aug 7 21:10:58.699165641 (4h44m12.286221892s) [upload=4897]: Starting upload Aug 7 21:10:58.699245781 (4h44m12.286302022s) [upload=4897]: Done starting upload Aug 7 21:10:58.699314669 (4h44m12.286370911s) [upload=4897]: Uploading segments... Aug 7 21:10:58.699373410 (4h44m12.286429651s) [upload=4897]: Got next segment. Inline: true Aug 7 21:10:58.699380724 (4h44m12.286436964s) [upload=4897]: Waiting for error group managing segments... Aug 7 21:10:58.699426338 (4h44m12.286482579s) [upload=4897]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:10:58.699432380 (4h44m12.28648862s) [upload=4897]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:58.699436066 (4h44m12.286492317s) [upload=4897]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:10:58.775720472 (4h44m12.362776712s) [upload=4897]: Done uploading segments... Aug 7 21:10:58.775736111 (4h44m12.362792352s) [upload=4897]: Upload finished. err: <nil> Aug 7 21:11:31.613155564 (4h44m45.200211785s) [upload=4908]: Starting upload Aug 7 21:11:31.613232477 (4h44m45.200288698s) [upload=4908]: Done starting upload Aug 7 21:11:31.613307799 (4h44m45.20036401s) [upload=4908]: Uploading segments... Aug 7 21:11:31.613347051 (4h44m45.200403272s) [upload=4908]: Got next segment. Inline: true Aug 7 21:11:31.613352932 (4h44m45.200409153s) [upload=4908]: Waiting for error group managing segments... Aug 7 21:11:31.613404179 (4h44m45.2004604s) [upload=4908]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:11:31.613410591 (4h44m45.200466842s) [upload=4908]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:31.613414508 (4h44m45.200470729s) [upload=4908]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.682826735 (4h44m45.269882977s) [upload=4908]: Done uploading segments... Aug 7 21:11:31.682853265 (4h44m45.269909476s) [upload=4908]: Upload finished. err: <nil> Aug 7 21:11:51.754258397 (4h45m5.341314618s) [upload=4915]: Starting upload Aug 7 21:11:51.754330331 (4h45m5.341386542s) [upload=4915]: Done starting upload Aug 7 21:11:51.754383972 (4h45m5.341440193s) [upload=4915]: Uploading segments... Aug 7 21:11:51.754417083 (4h45m5.341473294s) [upload=4915]: Got next segment. Inline: true Aug 7 21:11:51.754420630 (4h45m5.34147684s) [upload=4915]: Waiting for error group managing segments... Aug 7 21:11:51.754453131 (4h45m5.341509352s) [upload=4915]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:11:51.754457108 (4h45m5.341513329s) [upload=4915]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:51.754460725 (4h45m5.341516936s) [upload=4915]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:11:51.830658438 (4h45m5.417714649s) [upload=4915]: Done uploading segments... Aug 7 21:11:51.830673094 (4h45m5.417729315s) [upload=4915]: Upload finished. err: <nil> Aug 7 21:12:54.778494646 (4h46m8.365550856s) [upload=4932]: Starting upload Aug 7 21:12:54.778560719 (4h46m8.36561693s) [upload=4932]: Done starting upload Aug 7 21:12:54.778608638 (4h46m8.365664859s) [upload=4932]: Uploading segments... Aug 7 21:12:54.778637322 (4h46m8.365693543s) [upload=4932]: Got next segment. Inline: true Aug 7 21:12:54.778640999 (4h46m8.36569722s) [upload=4932]: Waiting for error group managing segments... Aug 7 21:12:54.778677527 (4h46m8.365733739s) [upload=4932]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:54.778681655 (4h46m8.365737876s) [upload=4932]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:54.778684771 (4h46m8.365740983s) [upload=4932]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:12:54.868239378 (4h46m8.455295599s) [upload=4932]: Done uploading segments... Aug 7 21:12:54.868253865 (4h46m8.455310086s) [upload=4932]: Upload finished. err: <nil> Aug 7 21:13:23.238906839 (4h46m36.82596306s) [upload=4942]: Starting upload Aug 7 21:13:23.238959356 (4h46m36.826015567s) [upload=4942]: Done starting upload Aug 7 21:13:23.239011013 (4h46m36.826067234s) [upload=4942]: Uploading segments... Aug 7 21:13:23.239039727 (4h46m36.826095937s) [upload=4942]: Got next segment. Inline: true Aug 7 21:13:23.239043483 (4h46m36.826099695s) [upload=4942]: Waiting for error group managing segments... Aug 7 21:13:23.239098526 (4h46m36.826154737s) [upload=4942]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:13:23.239104707 (4h46m36.826160919s) [upload=4942]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:23.239108474 (4h46m36.826164696s) [upload=4942]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:23.309357825 (4h46m36.896414127s) [upload=4942]: Done uploading segments... Aug 7 21:13:23.309383974 (4h46m36.896440216s) [upload=4942]: Upload finished. err: <nil> Aug 7 21:14:18.294119361 (4h47m31.881175572s) [upload=4957]: Starting upload Aug 7 21:14:18.294187468 (4h47m31.881243679s) [upload=4957]: Done starting upload Aug 7 21:14:18.294261516 (4h47m31.881317737s) [upload=4957]: Uploading segments... Aug 7 21:14:18.294305068 (4h47m31.881361289s) [upload=4957]: Got next segment. Inline: true Aug 7 21:14:18.294311409 (4h47m31.881367621s) [upload=4957]: Waiting for error group managing segments... Aug 7 21:14:18.294349491 (4h47m31.881405712s) [upload=4957]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:18.294354059 (4h47m31.88141028s) [upload=4957]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:18.294356643 (4h47m31.881412854s) [upload=4957]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:15:18.280461255 (4h48m31.867517496s) [upload=4957]: Done uploading segments... Aug 7 21:15:18.280484308 (4h48m31.867540559s) [upload=4957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:18.387536247 (4h48m31.974592488s) [upload=4974]: Starting upload Aug 7 21:15:18.387607770 (4h48m31.974664011s) [upload=4974]: Done starting upload Aug 7 21:15:18.387666109 (4h48m31.974722361s) [upload=4974]: Uploading segments... Aug 7 21:15:18.387706996 (4h48m31.974763236s) [upload=4974]: Got next segment. Inline: true Aug 7 21:15:18.387712766 (4h48m31.974769007s) [upload=4974]: Waiting for error group managing segments... Aug 7 21:15:18.387755796 (4h48m31.974812038s) [upload=4974]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:18.387761227 (4h48m31.974817468s) [upload=4974]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:18.387764853 (4h48m31.974821095s) [upload=4974]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:18.373970870 (4h49m31.961027121s) [upload=4974]: Done uploading segments... Aug 7 21:16:18.374035110 (4h49m31.961091352s) [upload=4974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:18.454541239 (4h49m32.04159748s) [upload=4989]: Starting upload Aug 7 21:16:18.454621288 (4h49m32.041677529s) [upload=4989]: Done starting upload Aug 7 21:16:18.454699754 (4h49m32.041756005s) [upload=4989]: Uploading segments... Aug 7 21:16:18.454758153 (4h49m32.041814395s) [upload=4989]: Got next segment. Inline: true Aug 7 21:16:18.454765147 (4h49m32.041821398s) [upload=4989]: Waiting for error group managing segments... Aug 7 21:16:18.454813777 (4h49m32.041870018s) [upload=4989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:18.454819818 (4h49m32.04187606s) [upload=4989]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:18.454823425 (4h49m32.041879666s) [upload=4989]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:18.435319201 (4h50m32.022375413s) [upload=4989]: Done uploading segments... Aug 7 21:17:18.435335041 (4h50m32.022391262s) [upload=4989]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:29.913551414 (4h50m43.500607635s) [upload=5005]: Starting upload Aug 7 21:17:29.913606246 (4h50m43.500662467s) [upload=5005]: Done starting upload Aug 7 21:17:29.913653174 (4h50m43.500709395s) [upload=5005]: Uploading segments... Aug 7 21:17:29.913680986 (4h50m43.500737207s) [upload=5005]: Got next segment. Inline: true Aug 7 21:17:29.913684613 (4h50m43.500740835s) [upload=5005]: Waiting for error group managing segments... Aug 7 21:17:29.913715249 (4h50m43.500771471s) [upload=5005]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:29.913719057 (4h50m43.500775278s) [upload=5005]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:29.913721331 (4h50m43.500777542s) [upload=5005]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:29.900370534 (4h51m43.487426745s) [upload=5005]: Done uploading segments... Aug 7 21:18:29.900390291 (4h51m43.487446542s) [upload=5005]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.000999839s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
32.911804ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:18:24 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 32.936261ms
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
9 out of 60 is failed (threshold=3) [.................................................XX..XXXXXXX] duration:
30.000670156s
FAIL (2025-08-07 21:18:19 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000686367s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
1m0.00105787s
FAIL (2025-08-07 21:18:10 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
21:13:24.168436171 (4h46m37.755492392s) [noise=true , address=104.36.82.13:4190 , piece_id=PVB3Y7KXU47SOQCZBKW4AUZ56XMLMJQ23WPM73BKHUMZKARYOCCA, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.168641956 (4h46m37.755698177s) [noise=true , address=45.91.116.10:4113 , piece_id=NJYFA7PI7F62J2CSEDJUTIWHY2LW3KBSLK76TKOVJNXC3RDV35KA, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.168845667 (4h46m37.755901877s) [noise=true , address=79.127.220.97:4396 , piece_id=HD3NDYSYACLTEX2OYRUDYNK5J2DREH3DTNSJSDO745Q6NCC7AXGA, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.169076598 (4h46m37.756132819s) [noise=true , address=104.36.82.44:4872 , piece_id=LM4MNB6MDHTN5MVJNGJOWX5Z5AKGICGK5JCLHBQBM5IR5PD3P2BQ, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.169509506 (4h46m37.756565727s) [noise=true , address=104.36.82.54:4183 , piece_id=PIBOQ56366UKNWMH43GSS6TW66GFPMSJOB3UQ3TKMEYEYPKVVFOQ, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.169710120 (4h46m37.756766341s) [noise=true , address=104.36.82.52:4108 , piece_id=AHQKVW323AG3TIIDCPX4HYFISVIY4V2476ZQM5GXLGLQD2PJ2KVA, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.260747004 (4h46m37.847803255s) [upload=4936]: Next returned nil segment
Aug 7 21:13:24.260784884 (4h46m37.847841125s) [upload=4936]: Waiting for error group managing segments...
Aug 7 21:13:24.282697478 (4h46m37.869753699s) [noise=true , address=104.36.82.36:4652 , piece_id=4S6GW6S5QMKXREUOZ4K7S77A5FIA5OGIS6MIPH3X5WFAIUYUTKBQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.286011234 (4h46m37.873067475s) [noise=true , address=45.91.116.10:4113 , piece_id=NJYFA7PI7F62J2CSEDJUTIWHY2LW3KBSLK76TKOVJNXC3RDV35KA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.288604064 (4h46m37.875660275s) [noise=true , address=104.36.82.23:4457 , piece_id=3LDUSHS23Z6ZBTDHVGVNUZRFA3H2XDC6B243ISKKCYJJG3PCRUBA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.289097404 (4h46m37.876153645s) [noise=true , address=104.36.82.44:4872 , piece_id=LM4MNB6MDHTN5MVJNGJOWX5Z5AKGICGK5JCLHBQBM5IR5PD3P2BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.303818073 (4h46m37.890874284s) [noise=true , address=104.36.82.20:4444 , piece_id=N3Z2NAQO5LHBPSABVMX4UBAAGUDDGDZCYGWX3NXWP6LUL5VJ74BA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.306609824 (4h46m37.893666035s) [noise=true , address=104.36.82.13:4190 , piece_id=PVB3Y7KXU47SOQCZBKW4AUZ56XMLMJQ23WPM73BKHUMZKARYOCCA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.307083188 (4h46m37.894139409s) [noise=true , address=104.36.82.46:4511 , piece_id=XTWLBKT2PX36GPDNTPPQ33HLJE2QMLNTVUMXVDEWJJO6IGC4HCXA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.333002740 (4h46m37.92005898s) [noise=true , address=104.36.82.11:4130 , piece_id=434EKDAN537AQFR342ZKLLENUIZAEK4PL2UTXDRJU7NJJ5HYHLUQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.335712287 (4h46m37.922768508s) [noise=true , address=79.127.220.97:4396 , piece_id=HD3NDYSYACLTEX2OYRUDYNK5J2DREH3DTNSJSDO745Q6NCC7AXGA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.364828819 (4h46m37.95188508s) [noise=true , address=104.36.82.54:4183 , piece_id=PIBOQ56366UKNWMH43GSS6TW66GFPMSJOB3UQ3TKMEYEYPKVVFOQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.415758456 (4h46m38.002814697s) [noise=true , address=104.36.82.37:4680 , piece_id=3LPKEUJ5HCZL4XY7YQOI2CROEHYNBFSZV6EUFDU2WV363FWQ644Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.427338251 (4h46m38.014394462s) [noise=true , address=104.36.82.43:4849 , piece_id=ZVVOBDRSIRLKPAOF23RMYRBXRNCNMQF3TJXYAW77ABOTGYUMXGHQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.490461549 (4h46m38.07751779s) [noise=true , address=79.127.201.215:4354 , piece_id=4VVIMMVTX6SCVPFAODIC7JS5PVULFS2SCXMKFJKA72NXKEK62K3Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.581523329 (4h46m38.16857954s) [noise=true , address=104.36.82.52:4108 , piece_id=AHQKVW323AG3TIIDCPX4HYFISVIY4V2476ZQM5GXLGLQD2PJ2KVA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.581552854 (4h46m38.168609065s) [seg_pos={0 0}, upload=4936]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:13:24.581774899 (4h46m38.16883112s) [noise=true , address=79.127.205.233:4643 , piece_id=IRDF6ZKJUCKCPSKAOCHW77H7O5ORLI43QULF2KWJ4O6SJTP373BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cVbdibJiX3T4fEUjeVBNNTTNaMb7m3QYDmxh17Dic5wVUQ9fx): protocol: expected piece hash; context canceled; EOF
Aug 7 21:13:24.581831154 (4h46m38.168887375s) [noise=true , address=104.36.82.62:4427 , piece_id=QMUVKOOD3THFKQAKBI3RQWH7JOP34MNNDOMKUXXBF4ZRVSYUJHJQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bx9fnLeXTPskzQB1Ad45qnq4CRivoGy7KXALY35dzVGF6S5Ep): protocol: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection
Aug 7 21:13:24.581866029 (4h46m38.168922239s) [noise=true , address=64.49.13.154:4933 , piece_id=HGHDC3TAI5HTPACRHKMJMLCMMF2AZOMT4DO5LIQEIECYYKCHMIRA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n5Mn8FNLXxFWtbbaMv8VuyxuXe6PiiMVtLETzm5aKfxhEykt3): protocol: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection
Aug 7 21:13:24.581904450 (4h46m38.168960661s) [noise=true , address=64.49.13.27:4963 , piece_id=MN6MBMOOPEIPZRW5R4NQM6AEE3AHYX33MIXOSIIZEGHDPJEKAOPA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq): protocol: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection
Aug 7 21:13:24.581925911 (4h46m38.168982132s) [seg_pos={0 0}, upload=4936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:13:24.582013845 (4h46m38.169070066s) [upload=4936]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:13:24.582018403 (4h46m38.169074624s) [upload=4936]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:15:00.743765287 (4h48m14.330821508s) [upload=4936]: Done uploading segments...
Aug 7 21:15:00.743795124 (4h48m14.330851345s) [upload=4936]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:15:00.812344761 (4h48m14.399400973s) [upload=4967]: Starting upload
Aug 7 21:15:00.812384405 (4h48m14.399440606s) [upload=4967]: Done starting upload
Aug 7 21:15:00.812509518 (4h48m14.399565729s) [upload=4967]: Uploading segments...
Aug 7 21:15:00.812592684 (4h48m14.399648895s) [upload=4967]: Got next segment. Inline: false
Aug 7 21:15:00.812634802 (4h48m14.399691023s) [upload=4967]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:15:00.812639241 (4h48m14.399695462s) [upload=4967]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:16:00.800009657 (4h49m14.387065878s) [upload=4967]: Next segment err: context deadline exceeded
Aug 7 21:16:00.800074909 (4h49m14.38713113s) [upload=4967]: Done uploading segments...
Aug 7 21:16:00.800088334 (4h49m14.387144545s) [upload=4967]: Upload finished. err: context deadline exceeded
Aug 7 21:16:09.662919397 (4h49m23.249975618s) [upload=4987]: Starting upload
Aug 7 21:16:09.663003895 (4h49m23.250060116s) [upload=4987]: Done starting upload
Aug 7 21:16:09.663197185 (4h49m23.250253426s) [upload=4987]: Uploading segments...
Aug 7 21:16:09.663304947 (4h49m23.250361248s) [upload=4987]: Got next segment. Inline: false
Aug 7 21:16:09.663375208 (4h49m23.250431459s) [upload=4987]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:16:09.663382832 (4h49m23.250439074s) [upload=4987]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:17:09.645661646 (4h50m23.232717887s) [upload=4987]: Next segment err: context deadline exceeded
Aug 7 21:17:09.645901063 (4h50m23.232957304s) [upload=4987]: Done uploading segments...
Aug 7 21:17:09.645976574 (4h50m23.233032815s) [upload=4987]: Upload finished. err: context deadline exceeded
Aug 7 21:17:10.545365898 (4h50m24.132422108s) [upload=5001]: Starting upload
Aug 7 21:17:10.545420008 (4h50m24.132476229s) [upload=5001]: Done starting upload
Aug 7 21:17:10.545476424 (4h50m24.132532645s) [upload=5001]: Uploading segments...
Aug 7 21:17:10.545636733 (4h50m24.132692944s) [upload=5001]: Got next segment. Inline: false
Aug 7 21:17:10.545661599 (4h50m24.13271782s) [upload=5001]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:17:10.545665476 (4h50m24.132721688s) [upload=5001]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:18:10.530620759 (4h51m24.11767701s) [upload=5001]: Next segment err: context deadline exceeded
Aug 7 21:18:10.530761893 (4h51m24.117818133s) [upload=5001]: Done uploading segments...
Aug 7 21:18:10.530782391 (4h51m24.117838632s) [upload=5001]: Upload finished. err: context deadline exceeded
Duration: 1m0.001077075s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration:
30.001058935s
FAIL (2025-08-07 21:18:08 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (4h44m2.623148821s) [upload=4895]: Upload finished. err: <nil> Aug 7 21:11:10.299224768 (4h44m23.886281009s) [upload=4903]: Starting upload Aug 7 21:11:10.299304937 (4h44m23.886361178s) [upload=4903]: Done starting upload Aug 7 21:11:10.299368445 (4h44m23.886424686s) [upload=4903]: Uploading segments... Aug 7 21:11:10.299409052 (4h44m23.886465292s) [upload=4903]: Got next segment. Inline: true Aug 7 21:11:10.299414652 (4h44m23.886470893s) [upload=4903]: Waiting for error group managing segments... Aug 7 21:11:10.299460468 (4h44m23.886516709s) [upload=4903]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:11:10.299465988 (4h44m23.886522229s) [upload=4903]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:10.299469444 (4h44m23.886525685s) [upload=4903]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:11:10.444467755 (4h44m24.031523996s) [upload=4903]: Done uploading segments... Aug 7 21:11:10.444485819 (4h44m24.03154206s) [upload=4903]: Upload finished. err: <nil> Aug 7 21:12:08.543001654 (4h45m22.130057896s) [upload=4919]: Starting upload Aug 7 21:12:08.543123191 (4h45m22.130179432s) [upload=4919]: Done starting upload Aug 7 21:12:08.543216246 (4h45m22.130272487s) [upload=4919]: Uploading segments... Aug 7 21:12:08.543261810 (4h45m22.130318051s) [upload=4919]: Got next segment. Inline: true Aug 7 21:12:08.543267651 (4h45m22.130323892s) [upload=4919]: Waiting for error group managing segments... Aug 7 21:12:08.543313246 (4h45m22.130369488s) [upload=4919]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:08.543319278 (4h45m22.130375519s) [upload=4919]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:08.543323075 (4h45m22.130379316s) [upload=4919]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:12:08.630954371 (4h45m22.218010612s) [upload=4919]: Done uploading segments... Aug 7 21:12:08.630979197 (4h45m22.218035449s) [upload=4919]: Upload finished. err: <nil> Aug 7 21:12:30.034507380 (4h45m43.62156362s) [upload=4926]: Starting upload Aug 7 21:12:30.034584302 (4h45m43.621640544s) [upload=4926]: Done starting upload Aug 7 21:12:30.034643523 (4h45m43.621699754s) [upload=4926]: Uploading segments... Aug 7 21:12:30.034681965 (4h45m43.621738206s) [upload=4926]: Got next segment. Inline: true Aug 7 21:12:30.034687645 (4h45m43.621743886s) [upload=4926]: Waiting for error group managing segments... Aug 7 21:12:30.034730766 (4h45m43.621787007s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:30.034735976 (4h45m43.621792207s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:30.034739642 (4h45m43.621795883s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:00.022476261 (4h46m13.609532503s) [upload=4926]: Done uploading segments... Aug 7 21:13:00.022497211 (4h46m13.609553452s) [upload=4926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:15.702096720 (4h46m29.289152941s) [upload=4939]: Starting upload Aug 7 21:13:15.702154518 (4h46m29.289210739s) [upload=4939]: Done starting upload Aug 7 21:13:15.702196716 (4h46m29.289252928s) [upload=4939]: Uploading segments... Aug 7 21:13:15.702224368 (4h46m29.289280589s) [upload=4939]: Got next segment. Inline: true Aug 7 21:13:15.702228446 (4h46m29.289284657s) [upload=4939]: Waiting for error group managing segments... Aug 7 21:13:15.702261458 (4h46m29.289317669s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:13:15.702265306 (4h46m29.289321527s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:15.702267880 (4h46m29.289324101s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.689162551 (4h46m59.276218792s) [upload=4939]: Done uploading segments... Aug 7 21:13:45.689208046 (4h46m59.276264277s) [upload=4939]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:13.522007492 (4h47m27.109063713s) [upload=4955]: Starting upload Aug 7 21:14:13.522102209 (4h47m27.10915843s) [upload=4955]: Done starting upload Aug 7 21:14:13.522154938 (4h47m27.109211179s) [upload=4955]: Uploading segments... Aug 7 21:14:13.522235488 (4h47m27.109291729s) [upload=4955]: Got next segment. Inline: true Aug 7 21:14:13.522242682 (4h47m27.109298922s) [upload=4955]: Waiting for error group managing segments... Aug 7 21:14:13.522293467 (4h47m27.109349708s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:13.522299518 (4h47m27.109355759s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:13.522303275 (4h47m27.109359516s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:14:43.507883903 (4h47m57.094940144s) [upload=4955]: Done uploading segments... Aug 7 21:14:43.507905914 (4h47m57.094962156s) [upload=4955]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:54.825501522 (4h48m8.412557734s) [upload=4964]: Starting upload Aug 7 21:14:54.825580730 (4h48m8.412636951s) [upload=4964]: Done starting upload Aug 7 21:14:54.825629411 (4h48m8.412685632s) [upload=4964]: Uploading segments... Aug 7 21:14:54.825660400 (4h48m8.41271661s) [upload=4964]: Got next segment. Inline: true Aug 7 21:14:54.825664547 (4h48m8.412720758s) [upload=4964]: Waiting for error group managing segments... Aug 7 21:14:54.825705873 (4h48m8.412762115s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:54.825711855 (4h48m8.412768076s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:54.825715662 (4h48m8.412771883s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:15:24.812961328 (4h48m38.400017569s) [upload=4964]: Done uploading segments... Aug 7 21:15:24.812988770 (4h48m38.40004501s) [upload=4964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:40.993035983 (4h48m54.580092194s) [upload=4978]: Starting upload Aug 7 21:15:40.993105403 (4h48m54.580161614s) [upload=4978]: Done starting upload Aug 7 21:15:40.993154163 (4h48m54.580210375s) [upload=4978]: Uploading segments... Aug 7 21:15:40.993189519 (4h48m54.580245741s) [upload=4978]: Got next segment. Inline: true Aug 7 21:15:40.993194319 (4h48m54.58025054s) [upload=4978]: Waiting for error group managing segments... Aug 7 21:15:40.993226930 (4h48m54.580283151s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:40.993231167 (4h48m54.580287379s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:40.993234264 (4h48m54.580290474s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:10.982081529 (4h49m24.569137771s) [upload=4978]: Done uploading segments... Aug 7 21:16:10.982104473 (4h49m24.569160713s) [upload=4978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:20.227399180 (4h49m33.814455401s) [upload=4990]: Starting upload Aug 7 21:16:20.227450334 (4h49m33.814506545s) [upload=4990]: Done starting upload Aug 7 21:16:20.227491060 (4h49m33.814547272s) [upload=4990]: Uploading segments... Aug 7 21:16:20.227516609 (4h49m33.81457283s) [upload=4990]: Got next segment. Inline: true Aug 7 21:16:20.227520536 (4h49m33.814576757s) [upload=4990]: Waiting for error group managing segments... Aug 7 21:16:20.227552426 (4h49m33.814608636s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:20.227556263 (4h49m33.814612484s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:20.227558527 (4h49m33.814614739s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:50.216081431 (4h50m3.803137652s) [upload=4990]: Done uploading segments... Aug 7 21:16:50.216097492 (4h50m3.803153713s) [upload=4990]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:57.322533914 (4h50m10.909590156s) [upload=4995]: Starting upload Aug 7 21:16:57.322609355 (4h50m10.909665607s) [upload=4995]: Done starting upload Aug 7 21:16:57.322672883 (4h50m10.909729125s) [upload=4995]: Uploading segments... Aug 7 21:16:57.322711135 (4h50m10.909767376s) [upload=4995]: Got next segment. Inline: true Aug 7 21:16:57.322716685 (4h50m10.909772927s) [upload=4995]: Waiting for error group managing segments... Aug 7 21:16:57.322761279 (4h50m10.90981752s) [upload=4995]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:57.322767100 (4h50m10.909823341s) [upload=4995]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:57.322770697 (4h50m10.909826938s) [upload=4995]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:27.310589172 (4h50m40.897645383s) [upload=4995]: Done uploading segments... Aug 7 21:17:27.310603289 (4h50m40.897659499s) [upload=4995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:38.458031408 (4h50m52.045087639s) [upload=5010]: Starting upload Aug 7 21:17:38.458135221 (4h50m52.045191452s) [upload=5010]: Done starting upload Aug 7 21:17:38.458199552 (4h50m52.045255784s) [upload=5010]: Uploading segments... Aug 7 21:17:38.458275263 (4h50m52.045331504s) [upload=5010]: Got next segment. Inline: true Aug 7 21:17:38.458283519 (4h50m52.04533976s) [upload=5010]: Waiting for error group managing segments... Aug 7 21:17:38.458329695 (4h50m52.045385926s) [upload=5010]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:38.458335165 (4h50m52.045391407s) [upload=5010]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:38.458338662 (4h50m52.045394893s) [upload=5010]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:08.444937554 (4h51m22.031993765s) [upload=5010]: Done uploading segments... Aug 7 21:18:08.444950428 (4h51m22.032006639s) [upload=5010]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.00108324s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.001150235s
FAIL (2025-08-07 21:18:04 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.001165662s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
1.014645574s
FAIL (2025-08-07 21:18:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment
Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments...
Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF
Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection
Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection
Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments...
Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil>
Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload
Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload
Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments...
Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false
Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests
Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments...
Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.014678165s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration:
30.000770816s
FAIL (2025-08-07 21:17:59 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
.82.9:4006 , piece_id=EH5ZUHUO55F7YYVD2TSWXDIUX65GY2VVVMBW5DOFRADL3WO3ZQGA, seg_pos={0 0}, upload=4938]: Uploading piece...
Aug 7 21:13:09.841271227 (4h46m23.428327438s) [upload=4938]: Next returned nil segment
Aug 7 21:13:09.841281055 (4h46m23.428337266s) [upload=4938]: Waiting for error group managing segments...
Aug 7 21:13:09.853631449 (4h46m23.440687659s) [noise=true , address=45.91.116.4:4937 , piece_id=HUS3E4LN6CYOXDY2DIEK4OATMZPZADDVIQO3CM3L5QEMB567P3KA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.854204258 (4h46m23.441260499s) [noise=true , address=45.91.116.12:4225 , piece_id=EPHX3ZYTEBJUTQP56HQSL4GPW5SBMC7KDNSR6BD7CRQTMCDUZ2DQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.854658546 (4h46m23.441714757s) [noise=true , address=45.91.116.14:4191 , piece_id=23HFDLS3OMLHX4EBBB3N6ORXPOJGZ4TSLXD622Z5QW57I7F33YSQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.855360527 (4h46m23.442416778s) [noise=true , address=77.90.31.18:4807 , piece_id=GYJGSHU2PC4DM5VA6SLHGM7S45KOLLBLBTMKD6LT5LMAEMHKGK5Q, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.855562183 (4h46m23.442618394s) [noise=true , address=104.36.82.13:4185 , piece_id=JRWMK5KTTCJIL5JJ4YVJ3GHS6BYCZP3CRWM4AN4ZL2UU4HJADBSQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.856428770 (4h46m23.443485021s) [noise=true , address=77.90.31.19:4841 , piece_id=AKJV4UNYIXKLSI4Y5STPWBXJ3AFZERBSVCIER53F23KL5WABQFTQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.856590482 (4h46m23.443646684s) [noise=true , address=104.36.82.2:4729 , piece_id=KQOVEHNICVG5FWPV2B7CP3U6SMP56T7PRNZOG65QDAX2MP4SEN7Q, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.858568354 (4h46m23.445624565s) [noise=true , address=45.91.116.13:4155 , piece_id=BXSMK5DW67MELF2KN6CIDHJ5HOSF7MZBRVSA7WQ45UV4SLI5XHWQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.858850201 (4h46m23.445906441s) [noise=true , address=77.90.31.16:4733 , piece_id=EMDHMK2PGYUT3SUXOVNEKLRHOH3XY4WNXON4OXTERRMZI43GCROA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.859375651 (4h46m23.446431862s) [noise=true , address=77.90.31.17:4772 , piece_id=QUJ2AZMAXL2FFEWVASVIN734LGSMBFRXHX5KORJLQNGIXNN64ZVA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.859727598 (4h46m23.446783819s) [noise=true , address=104.36.82.13:4187 , piece_id=J3E4KCSQZDYKXQMOTRSUUUIU2SZIRFBM7WD32FCFJL7V7DILAQXA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.862181970 (4h46m23.44923821s) [noise=true , address=104.36.82.12:4151 , piece_id=OIWOXCCCVAW3B5ISSLNWASC6TD2WSWC4J3EJMF554OHVQMCZUIWQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.862286064 (4h46m23.449342285s) [noise=true , address=104.36.82.7:4910 , piece_id=HFOIPKET4SKDQ4LRX54T7VOA7V2Y46MZ6RH5DED63ZXTTDYZTHQA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.862295481 (4h46m23.449351692s) [seg_pos={0 0}, upload=4938]: Segment reached optimal threshold of 13 pieces.
Aug 7 21:13:09.862540559 (4h46m23.44959677s) [noise=true , address=104.36.82.9:4006 , piece_id=EH5ZUHUO55F7YYVD2TSWXDIUX65GY2VVVMBW5DOFRADL3WO3ZQGA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 5.161.89.252:35382->104.36.82.9:4006: use of closed network connection; write tcp 5.161.89.252:35382->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:35382->104.36.82.9:4006: use of closed network connection
Aug 7 21:13:09.862594711 (4h46m23.449650932s) [noise=true , address=104.36.82.5:4839 , piece_id=JFXBI2UEF5IPMF3NFKEX4CBRRNCIBJCSLLSVKCTFRTYRD3U4TLLQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 5.161.89.252:52042->104.36.82.5:4839: use of closed network connection; write tcp 5.161.89.252:52042->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:52042->104.36.82.5:4839: use of closed network connection
Aug 7 21:13:09.862660843 (4h46m23.449717085s) [noise=true , address=104.36.82.15:4259 , piece_id=AZDG4LGBEPEVUA4KZH24JWD2BCFEUA2IGBFWC57N4D5WVXBJ27PA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 5.161.89.252:8524->104.36.82.15:4259: use of closed network connection; write tcp 5.161.89.252:8524->104.36.82.15:4259: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:8524->104.36.82.15:4259: use of closed network connection
Aug 7 21:13:09.862727478 (4h46m23.449783718s) [seg_pos={0 0}, upload=4938]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 21:13:09.862892626 (4h46m23.449948867s) [upload=4938]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:13:09.862899970 (4h46m23.449956211s) [upload=4938]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:13:09.912447959 (4h46m23.499504169s) [upload=4938]: Done uploading segments...
Aug 7 21:13:09.912480760 (4h46m23.499536981s) [upload=4938]: Upload finished. err: <nil>
Aug 7 21:14:19.537010001 (4h47m33.124066212s) [upload=4958]: Starting upload
Aug 7 21:14:19.537096253 (4h47m33.124152504s) [upload=4958]: Done starting upload
Aug 7 21:14:19.537202491 (4h47m33.124258732s) [upload=4958]: Uploading segments...
Aug 7 21:14:19.537385743 (4h47m33.124441964s) [upload=4958]: Got next segment. Inline: false
Aug 7 21:14:19.537463918 (4h47m33.124520139s) [upload=4958]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:14:19.537469729 (4h47m33.12452595s) [upload=4958]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:14:49.522961209 (4h48m3.11001742s) [upload=4958]: Next segment err: context deadline exceeded
Aug 7 21:14:49.523117610 (4h48m3.110173821s) [upload=4958]: Done uploading segments...
Aug 7 21:14:49.523153057 (4h48m3.110209278s) [upload=4958]: Upload finished. err: context deadline exceeded
Aug 7 21:15:15.778236986 (4h48m29.365293237s) [upload=4971]: Starting upload
Aug 7 21:15:15.778363202 (4h48m29.365419443s) [upload=4971]: Done starting upload
Aug 7 21:15:15.778550181 (4h48m29.365606442s) [upload=4971]: Uploading segments...
Aug 7 21:15:15.778800839 (4h48m29.36585709s) [upload=4971]: Got next segment. Inline: false
Aug 7 21:15:15.778897088 (4h48m29.36595334s) [upload=4971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:15:15.778908650 (4h48m29.365964891s) [upload=4971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:15:45.754152630 (4h48m59.341208851s) [upload=4971]: Next segment err: context deadline exceeded
Aug 7 21:15:45.754250592 (4h48m59.341306803s) [upload=4971]: Done uploading segments...
Aug 7 21:15:45.754272974 (4h48m59.341329186s) [upload=4971]: Upload finished. err: context deadline exceeded
Aug 7 21:15:54.388794729 (4h49m7.975850951s) [upload=4981]: Starting upload
Aug 7 21:15:54.388846236 (4h49m7.975902457s) [upload=4981]: Done starting upload
Aug 7 21:15:54.388936303 (4h49m7.975992524s) [upload=4981]: Uploading segments...
Aug 7 21:15:54.389106811 (4h49m7.976163032s) [upload=4981]: Got next segment. Inline: false
Aug 7 21:15:54.389136207 (4h49m7.976192428s) [upload=4981]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:15:54.389140785 (4h49m7.976197006s) [upload=4981]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:16:24.377635868 (4h49m37.96469211s) [upload=4981]: Next segment err: context deadline exceeded
Aug 7 21:16:24.377706209 (4h49m37.96476246s) [upload=4981]: Done uploading segments...
Aug 7 21:16:24.377728742 (4h49m37.964784992s) [upload=4981]: Upload finished. err: context deadline exceeded
Aug 7 21:16:48.671509340 (4h50m2.258565551s) [upload=4993]: Starting upload
Aug 7 21:16:48.671596032 (4h50m2.258652243s) [upload=4993]: Done starting upload
Aug 7 21:16:48.671714142 (4h50m2.258770383s) [upload=4993]: Uploading segments...
Aug 7 21:16:48.671899698 (4h50m2.25895594s) [upload=4993]: Got next segment. Inline: false
Aug 7 21:16:48.671982002 (4h50m2.259038243s) [upload=4993]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:16:48.671989065 (4h50m2.259045306s) [upload=4993]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:17:18.658253100 (4h50m32.245309321s) [upload=4993]: Next segment err: context deadline exceeded
Aug 7 21:17:18.658397720 (4h50m32.245453941s) [upload=4993]: Done uploading segments...
Aug 7 21:17:18.658431633 (4h50m32.245487854s) [upload=4993]: Upload finished. err: context deadline exceeded
Aug 7 21:17:29.086779152 (4h50m42.673835373s) [upload=5004]: Starting upload
Aug 7 21:17:29.086833233 (4h50m42.673889444s) [upload=5004]: Done starting upload
Aug 7 21:17:29.086916819 (4h50m42.67397304s) [upload=5004]: Uploading segments...
Aug 7 21:17:29.087068823 (4h50m42.674125034s) [upload=5004]: Got next segment. Inline: false
Aug 7 21:17:29.087119998 (4h50m42.674176219s) [upload=5004]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:17:29.087124536 (4h50m42.674180748s) [upload=5004]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:17:59.072413634 (4h51m12.659469875s) [upload=5004]: Next segment err: context deadline exceeded
Aug 7 21:17:59.072514252 (4h51m12.659570503s) [upload=5004]: Done uploading segments...
Aug 7 21:17:59.072534319 (4h51m12.659590571s) [upload=5004]: Upload finished. err: context deadline exceeded
Duration: 30.00078927s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
30.001216165s
FAIL (2025-08-07 21:17:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001227518s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration:
30.001247535s
FAIL (2025-08-07 21:17:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001266729s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [......................................................XX.XXX] duration:
1m0.000493193s
FAIL (2025-08-07 21:17:51 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000506988s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
8 out of 60 is failed (threshold=3) [..................................................XX..XXXXXX] duration:
30.00052132s
FAIL (2025-08-07 21:17:49 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000545607s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
563.883791ms
FAIL (2025-08-07 21:17:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment
Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments...
Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF
Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection
Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection
Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments...
Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil>
Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload
Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload
Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments...
Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false
Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests
Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments...
Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests
Duration: 563.915331ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
5m41.035929323s
FAIL (2025-08-07 21:17:34 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5m41.035953329s
core
uplink
us1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
25.048899ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:17:32 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 25.065679ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
2m0.001119917s remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:17:31 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 2m0.001144244s
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration:
30.000480121s
FAIL (2025-08-07 21:17:27 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g segments... Aug 7 21:09:58.687873194 (4h43m12.274929435s) [upload=4880]: Upload finished. err: <nil> Aug 7 21:10:48.980309603 (4h44m2.567365814s) [upload=4895]: Starting upload Aug 7 21:10:48.980351321 (4h44m2.567407542s) [upload=4895]: Done starting upload Aug 7 21:10:48.980412926 (4h44m2.567469137s) [upload=4895]: Uploading segments... Aug 7 21:10:48.980468068 (4h44m2.567524309s) [upload=4895]: Got next segment. Inline: true Aug 7 21:10:48.980474410 (4h44m2.567530621s) [upload=4895]: Waiting for error group managing segments... Aug 7 21:10:48.980519745 (4h44m2.567575966s) [upload=4895]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:10:48.980525596 (4h44m2.567581807s) [upload=4895]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:48.980529363 (4h44m2.567585584s) [upload=4895]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:10:49.036078074 (4h44m2.623134295s) [upload=4895]: Done uploading segments... Aug 7 21:10:49.036092611 (4h44m2.623148821s) [upload=4895]: Upload finished. err: <nil> Aug 7 21:11:10.299224768 (4h44m23.886281009s) [upload=4903]: Starting upload Aug 7 21:11:10.299304937 (4h44m23.886361178s) [upload=4903]: Done starting upload Aug 7 21:11:10.299368445 (4h44m23.886424686s) [upload=4903]: Uploading segments... Aug 7 21:11:10.299409052 (4h44m23.886465292s) [upload=4903]: Got next segment. Inline: true Aug 7 21:11:10.299414652 (4h44m23.886470893s) [upload=4903]: Waiting for error group managing segments... Aug 7 21:11:10.299460468 (4h44m23.886516709s) [upload=4903]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:11:10.299465988 (4h44m23.886522229s) [upload=4903]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:10.299469444 (4h44m23.886525685s) [upload=4903]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:11:10.444467755 (4h44m24.031523996s) [upload=4903]: Done uploading segments... Aug 7 21:11:10.444485819 (4h44m24.03154206s) [upload=4903]: Upload finished. err: <nil> Aug 7 21:12:08.543001654 (4h45m22.130057896s) [upload=4919]: Starting upload Aug 7 21:12:08.543123191 (4h45m22.130179432s) [upload=4919]: Done starting upload Aug 7 21:12:08.543216246 (4h45m22.130272487s) [upload=4919]: Uploading segments... Aug 7 21:12:08.543261810 (4h45m22.130318051s) [upload=4919]: Got next segment. Inline: true Aug 7 21:12:08.543267651 (4h45m22.130323892s) [upload=4919]: Waiting for error group managing segments... Aug 7 21:12:08.543313246 (4h45m22.130369488s) [upload=4919]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:08.543319278 (4h45m22.130375519s) [upload=4919]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:08.543323075 (4h45m22.130379316s) [upload=4919]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:12:08.630954371 (4h45m22.218010612s) [upload=4919]: Done uploading segments... Aug 7 21:12:08.630979197 (4h45m22.218035449s) [upload=4919]: Upload finished. err: <nil> Aug 7 21:12:30.034507380 (4h45m43.62156362s) [upload=4926]: Starting upload Aug 7 21:12:30.034584302 (4h45m43.621640544s) [upload=4926]: Done starting upload Aug 7 21:12:30.034643523 (4h45m43.621699754s) [upload=4926]: Uploading segments... Aug 7 21:12:30.034681965 (4h45m43.621738206s) [upload=4926]: Got next segment. Inline: true Aug 7 21:12:30.034687645 (4h45m43.621743886s) [upload=4926]: Waiting for error group managing segments... Aug 7 21:12:30.034730766 (4h45m43.621787007s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:30.034735976 (4h45m43.621792207s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:30.034739642 (4h45m43.621795883s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:00.022476261 (4h46m13.609532503s) [upload=4926]: Done uploading segments... Aug 7 21:13:00.022497211 (4h46m13.609553452s) [upload=4926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:15.702096720 (4h46m29.289152941s) [upload=4939]: Starting upload Aug 7 21:13:15.702154518 (4h46m29.289210739s) [upload=4939]: Done starting upload Aug 7 21:13:15.702196716 (4h46m29.289252928s) [upload=4939]: Uploading segments... Aug 7 21:13:15.702224368 (4h46m29.289280589s) [upload=4939]: Got next segment. Inline: true Aug 7 21:13:15.702228446 (4h46m29.289284657s) [upload=4939]: Waiting for error group managing segments... Aug 7 21:13:15.702261458 (4h46m29.289317669s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:13:15.702265306 (4h46m29.289321527s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:15.702267880 (4h46m29.289324101s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.689162551 (4h46m59.276218792s) [upload=4939]: Done uploading segments... Aug 7 21:13:45.689208046 (4h46m59.276264277s) [upload=4939]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:13.522007492 (4h47m27.109063713s) [upload=4955]: Starting upload Aug 7 21:14:13.522102209 (4h47m27.10915843s) [upload=4955]: Done starting upload Aug 7 21:14:13.522154938 (4h47m27.109211179s) [upload=4955]: Uploading segments... Aug 7 21:14:13.522235488 (4h47m27.109291729s) [upload=4955]: Got next segment. Inline: true Aug 7 21:14:13.522242682 (4h47m27.109298922s) [upload=4955]: Waiting for error group managing segments... Aug 7 21:14:13.522293467 (4h47m27.109349708s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:13.522299518 (4h47m27.109355759s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:13.522303275 (4h47m27.109359516s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:14:43.507883903 (4h47m57.094940144s) [upload=4955]: Done uploading segments... Aug 7 21:14:43.507905914 (4h47m57.094962156s) [upload=4955]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:54.825501522 (4h48m8.412557734s) [upload=4964]: Starting upload Aug 7 21:14:54.825580730 (4h48m8.412636951s) [upload=4964]: Done starting upload Aug 7 21:14:54.825629411 (4h48m8.412685632s) [upload=4964]: Uploading segments... Aug 7 21:14:54.825660400 (4h48m8.41271661s) [upload=4964]: Got next segment. Inline: true Aug 7 21:14:54.825664547 (4h48m8.412720758s) [upload=4964]: Waiting for error group managing segments... Aug 7 21:14:54.825705873 (4h48m8.412762115s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:54.825711855 (4h48m8.412768076s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:54.825715662 (4h48m8.412771883s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:15:24.812961328 (4h48m38.400017569s) [upload=4964]: Done uploading segments... Aug 7 21:15:24.812988770 (4h48m38.40004501s) [upload=4964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:40.993035983 (4h48m54.580092194s) [upload=4978]: Starting upload Aug 7 21:15:40.993105403 (4h48m54.580161614s) [upload=4978]: Done starting upload Aug 7 21:15:40.993154163 (4h48m54.580210375s) [upload=4978]: Uploading segments... Aug 7 21:15:40.993189519 (4h48m54.580245741s) [upload=4978]: Got next segment. Inline: true Aug 7 21:15:40.993194319 (4h48m54.58025054s) [upload=4978]: Waiting for error group managing segments... Aug 7 21:15:40.993226930 (4h48m54.580283151s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:40.993231167 (4h48m54.580287379s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:40.993234264 (4h48m54.580290474s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:10.982081529 (4h49m24.569137771s) [upload=4978]: Done uploading segments... Aug 7 21:16:10.982104473 (4h49m24.569160713s) [upload=4978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:20.227399180 (4h49m33.814455401s) [upload=4990]: Starting upload Aug 7 21:16:20.227450334 (4h49m33.814506545s) [upload=4990]: Done starting upload Aug 7 21:16:20.227491060 (4h49m33.814547272s) [upload=4990]: Uploading segments... Aug 7 21:16:20.227516609 (4h49m33.81457283s) [upload=4990]: Got next segment. Inline: true Aug 7 21:16:20.227520536 (4h49m33.814576757s) [upload=4990]: Waiting for error group managing segments... Aug 7 21:16:20.227552426 (4h49m33.814608636s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:20.227556263 (4h49m33.814612484s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:20.227558527 (4h49m33.814614739s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:50.216081431 (4h50m3.803137652s) [upload=4990]: Done uploading segments... Aug 7 21:16:50.216097492 (4h50m3.803153713s) [upload=4990]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:57.322533914 (4h50m10.909590156s) [upload=4995]: Starting upload Aug 7 21:16:57.322609355 (4h50m10.909665607s) [upload=4995]: Done starting upload Aug 7 21:16:57.322672883 (4h50m10.909729125s) [upload=4995]: Uploading segments... Aug 7 21:16:57.322711135 (4h50m10.909767376s) [upload=4995]: Got next segment. Inline: true Aug 7 21:16:57.322716685 (4h50m10.909772927s) [upload=4995]: Waiting for error group managing segments... Aug 7 21:16:57.322761279 (4h50m10.90981752s) [upload=4995]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:57.322767100 (4h50m10.909823341s) [upload=4995]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:57.322770697 (4h50m10.909826938s) [upload=4995]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:27.310589172 (4h50m40.897645383s) [upload=4995]: Done uploading segments... Aug 7 21:17:27.310603289 (4h50m40.897659499s) [upload=4995]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.000493566s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
30.001186649s
FAIL (2025-08-07 21:17:18 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
36212920 (4h46m23.323269172s) [noise=true , address=104.36.82.2:4729 , piece_id=KQOVEHNICVG5FWPV2B7CP3U6SMP56T7PRNZOG65QDAX2MP4SEN7Q, seg_pos={0 0}, upload=4938]: Uploading piece...
Aug 7 21:13:09.736236384 (4h46m23.323292625s) [noise=true , address=77.90.31.18:4807 , piece_id=GYJGSHU2PC4DM5VA6SLHGM7S45KOLLBLBTMKD6LT5LMAEMHKGK5Q, seg_pos={0 0}, upload=4938]: Uploading piece...
Aug 7 21:13:09.736629768 (4h46m23.323686009s) [noise=true , address=104.36.82.13:4187 , piece_id=J3E4KCSQZDYKXQMOTRSUUUIU2SZIRFBM7WD32FCFJL7V7DILAQXA, seg_pos={0 0}, upload=4938]: Uploading piece...
Aug 7 21:13:09.736782593 (4h46m23.323838835s) [noise=true , address=45.91.116.12:4225 , piece_id=EPHX3ZYTEBJUTQP56HQSL4GPW5SBMC7KDNSR6BD7CRQTMCDUZ2DQ, seg_pos={0 0}, upload=4938]: Uploading piece...
Aug 7 21:13:09.737971863 (4h46m23.325028114s) [noise=true , address=104.36.82.9:4006 , piece_id=EH5ZUHUO55F7YYVD2TSWXDIUX65GY2VVVMBW5DOFRADL3WO3ZQGA, seg_pos={0 0}, upload=4938]: Uploading piece...
Aug 7 21:13:09.841271227 (4h46m23.428327438s) [upload=4938]: Next returned nil segment
Aug 7 21:13:09.841281055 (4h46m23.428337266s) [upload=4938]: Waiting for error group managing segments...
Aug 7 21:13:09.853631449 (4h46m23.440687659s) [noise=true , address=45.91.116.4:4937 , piece_id=HUS3E4LN6CYOXDY2DIEK4OATMZPZADDVIQO3CM3L5QEMB567P3KA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.854204258 (4h46m23.441260499s) [noise=true , address=45.91.116.12:4225 , piece_id=EPHX3ZYTEBJUTQP56HQSL4GPW5SBMC7KDNSR6BD7CRQTMCDUZ2DQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.854658546 (4h46m23.441714757s) [noise=true , address=45.91.116.14:4191 , piece_id=23HFDLS3OMLHX4EBBB3N6ORXPOJGZ4TSLXD622Z5QW57I7F33YSQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.855360527 (4h46m23.442416778s) [noise=true , address=77.90.31.18:4807 , piece_id=GYJGSHU2PC4DM5VA6SLHGM7S45KOLLBLBTMKD6LT5LMAEMHKGK5Q, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.855562183 (4h46m23.442618394s) [noise=true , address=104.36.82.13:4185 , piece_id=JRWMK5KTTCJIL5JJ4YVJ3GHS6BYCZP3CRWM4AN4ZL2UU4HJADBSQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.856428770 (4h46m23.443485021s) [noise=true , address=77.90.31.19:4841 , piece_id=AKJV4UNYIXKLSI4Y5STPWBXJ3AFZERBSVCIER53F23KL5WABQFTQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.856590482 (4h46m23.443646684s) [noise=true , address=104.36.82.2:4729 , piece_id=KQOVEHNICVG5FWPV2B7CP3U6SMP56T7PRNZOG65QDAX2MP4SEN7Q, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.858568354 (4h46m23.445624565s) [noise=true , address=45.91.116.13:4155 , piece_id=BXSMK5DW67MELF2KN6CIDHJ5HOSF7MZBRVSA7WQ45UV4SLI5XHWQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.858850201 (4h46m23.445906441s) [noise=true , address=77.90.31.16:4733 , piece_id=EMDHMK2PGYUT3SUXOVNEKLRHOH3XY4WNXON4OXTERRMZI43GCROA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.859375651 (4h46m23.446431862s) [noise=true , address=77.90.31.17:4772 , piece_id=QUJ2AZMAXL2FFEWVASVIN734LGSMBFRXHX5KORJLQNGIXNN64ZVA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.859727598 (4h46m23.446783819s) [noise=true , address=104.36.82.13:4187 , piece_id=J3E4KCSQZDYKXQMOTRSUUUIU2SZIRFBM7WD32FCFJL7V7DILAQXA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.862181970 (4h46m23.44923821s) [noise=true , address=104.36.82.12:4151 , piece_id=OIWOXCCCVAW3B5ISSLNWASC6TD2WSWC4J3EJMF554OHVQMCZUIWQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.862286064 (4h46m23.449342285s) [noise=true , address=104.36.82.7:4910 , piece_id=HFOIPKET4SKDQ4LRX54T7VOA7V2Y46MZ6RH5DED63ZXTTDYZTHQA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil>
Aug 7 21:13:09.862295481 (4h46m23.449351692s) [seg_pos={0 0}, upload=4938]: Segment reached optimal threshold of 13 pieces.
Aug 7 21:13:09.862540559 (4h46m23.44959677s) [noise=true , address=104.36.82.9:4006 , piece_id=EH5ZUHUO55F7YYVD2TSWXDIUX65GY2VVVMBW5DOFRADL3WO3ZQGA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 5.161.89.252:35382->104.36.82.9:4006: use of closed network connection; write tcp 5.161.89.252:35382->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:35382->104.36.82.9:4006: use of closed network connection
Aug 7 21:13:09.862594711 (4h46m23.449650932s) [noise=true , address=104.36.82.5:4839 , piece_id=JFXBI2UEF5IPMF3NFKEX4CBRRNCIBJCSLLSVKCTFRTYRD3U4TLLQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 5.161.89.252:52042->104.36.82.5:4839: use of closed network connection; write tcp 5.161.89.252:52042->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:52042->104.36.82.5:4839: use of closed network connection
Aug 7 21:13:09.862660843 (4h46m23.449717085s) [noise=true , address=104.36.82.15:4259 , piece_id=AZDG4LGBEPEVUA4KZH24JWD2BCFEUA2IGBFWC57N4D5WVXBJ27PA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 5.161.89.252:8524->104.36.82.15:4259: use of closed network connection; write tcp 5.161.89.252:8524->104.36.82.15:4259: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:8524->104.36.82.15:4259: use of closed network connection
Aug 7 21:13:09.862727478 (4h46m23.449783718s) [seg_pos={0 0}, upload=4938]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 21:13:09.862892626 (4h46m23.449948867s) [upload=4938]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:13:09.862899970 (4h46m23.449956211s) [upload=4938]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:13:09.912447959 (4h46m23.499504169s) [upload=4938]: Done uploading segments...
Aug 7 21:13:09.912480760 (4h46m23.499536981s) [upload=4938]: Upload finished. err: <nil>
Aug 7 21:14:19.537010001 (4h47m33.124066212s) [upload=4958]: Starting upload
Aug 7 21:14:19.537096253 (4h47m33.124152504s) [upload=4958]: Done starting upload
Aug 7 21:14:19.537202491 (4h47m33.124258732s) [upload=4958]: Uploading segments...
Aug 7 21:14:19.537385743 (4h47m33.124441964s) [upload=4958]: Got next segment. Inline: false
Aug 7 21:14:19.537463918 (4h47m33.124520139s) [upload=4958]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:14:19.537469729 (4h47m33.12452595s) [upload=4958]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:14:49.522961209 (4h48m3.11001742s) [upload=4958]: Next segment err: context deadline exceeded
Aug 7 21:14:49.523117610 (4h48m3.110173821s) [upload=4958]: Done uploading segments...
Aug 7 21:14:49.523153057 (4h48m3.110209278s) [upload=4958]: Upload finished. err: context deadline exceeded
Aug 7 21:15:15.778236986 (4h48m29.365293237s) [upload=4971]: Starting upload
Aug 7 21:15:15.778363202 (4h48m29.365419443s) [upload=4971]: Done starting upload
Aug 7 21:15:15.778550181 (4h48m29.365606442s) [upload=4971]: Uploading segments...
Aug 7 21:15:15.778800839 (4h48m29.36585709s) [upload=4971]: Got next segment. Inline: false
Aug 7 21:15:15.778897088 (4h48m29.36595334s) [upload=4971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:15:15.778908650 (4h48m29.365964891s) [upload=4971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:15:45.754152630 (4h48m59.341208851s) [upload=4971]: Next segment err: context deadline exceeded
Aug 7 21:15:45.754250592 (4h48m59.341306803s) [upload=4971]: Done uploading segments...
Aug 7 21:15:45.754272974 (4h48m59.341329186s) [upload=4971]: Upload finished. err: context deadline exceeded
Aug 7 21:15:54.388794729 (4h49m7.975850951s) [upload=4981]: Starting upload
Aug 7 21:15:54.388846236 (4h49m7.975902457s) [upload=4981]: Done starting upload
Aug 7 21:15:54.388936303 (4h49m7.975992524s) [upload=4981]: Uploading segments...
Aug 7 21:15:54.389106811 (4h49m7.976163032s) [upload=4981]: Got next segment. Inline: false
Aug 7 21:15:54.389136207 (4h49m7.976192428s) [upload=4981]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:15:54.389140785 (4h49m7.976197006s) [upload=4981]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:16:24.377635868 (4h49m37.96469211s) [upload=4981]: Next segment err: context deadline exceeded
Aug 7 21:16:24.377706209 (4h49m37.96476246s) [upload=4981]: Done uploading segments...
Aug 7 21:16:24.377728742 (4h49m37.964784992s) [upload=4981]: Upload finished. err: context deadline exceeded
Aug 7 21:16:48.671509340 (4h50m2.258565551s) [upload=4993]: Starting upload
Aug 7 21:16:48.671596032 (4h50m2.258652243s) [upload=4993]: Done starting upload
Aug 7 21:16:48.671714142 (4h50m2.258770383s) [upload=4993]: Uploading segments...
Aug 7 21:16:48.671899698 (4h50m2.25895594s) [upload=4993]: Got next segment. Inline: false
Aug 7 21:16:48.671982002 (4h50m2.259038243s) [upload=4993]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:16:48.671989065 (4h50m2.259045306s) [upload=4993]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:17:18.658253100 (4h50m32.245309321s) [upload=4993]: Next segment err: context deadline exceeded
Aug 7 21:17:18.658397720 (4h50m32.245453941s) [upload=4993]: Done uploading segments...
Aug 7 21:17:18.658431633 (4h50m32.245487854s) [upload=4993]: Upload finished. err: context deadline exceeded
Duration: 30.001207268s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
1m0.000877159s
FAIL (2025-08-07 21:17:18 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad=4858]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:08:39.738955514 (4h41m53.326011754s) [upload=4858]: Done uploading segments... Aug 7 21:08:39.738974128 (4h41m53.32603037s) [upload=4858]: Upload finished. err: <nil> Aug 7 21:09:18.312602069 (4h42m31.899658319s) [upload=4869]: Starting upload Aug 7 21:09:18.312710330 (4h42m31.899766582s) [upload=4869]: Done starting upload Aug 7 21:09:18.312805559 (4h42m31.89986181s) [upload=4869]: Uploading segments... Aug 7 21:09:18.312872103 (4h42m31.899928344s) [upload=4869]: Got next segment. Inline: true Aug 7 21:09:18.312879376 (4h42m31.899935618s) [upload=4869]: Waiting for error group managing segments... Aug 7 21:09:18.312928889 (4h42m31.89998514s) [upload=4869]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:09:18.312934650 (4h42m31.8999909s) [upload=4869]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:18.312938246 (4h42m31.899994497s) [upload=4869]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:09:18.367751803 (4h42m31.954808044s) [upload=4869]: Done uploading segments... Aug 7 21:09:18.367774827 (4h42m31.954831078s) [upload=4869]: Upload finished. err: <nil> Aug 7 21:10:26.543295840 (4h43m40.130352072s) [upload=4887]: Starting upload Aug 7 21:10:26.543341496 (4h43m40.130397717s) [upload=4887]: Done starting upload Aug 7 21:10:26.543385698 (4h43m40.130441919s) [upload=4887]: Uploading segments... Aug 7 21:10:26.543414672 (4h43m40.130470883s) [upload=4887]: Got next segment. Inline: true Aug 7 21:10:26.543418339 (4h43m40.13047455s) [upload=4887]: Waiting for error group managing segments... Aug 7 21:10:26.543446882 (4h43m40.130503103s) [upload=4887]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:10:26.543450790 (4h43m40.130507011s) [upload=4887]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:26.543453054 (4h43m40.130509265s) [upload=4887]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:10:26.611321150 (4h43m40.198377371s) [upload=4887]: Done uploading segments... Aug 7 21:10:26.611339073 (4h43m40.198395294s) [upload=4887]: Upload finished. err: <nil> Aug 7 21:10:58.699165641 (4h44m12.286221892s) [upload=4897]: Starting upload Aug 7 21:10:58.699245781 (4h44m12.286302022s) [upload=4897]: Done starting upload Aug 7 21:10:58.699314669 (4h44m12.286370911s) [upload=4897]: Uploading segments... Aug 7 21:10:58.699373410 (4h44m12.286429651s) [upload=4897]: Got next segment. Inline: true Aug 7 21:10:58.699380724 (4h44m12.286436964s) [upload=4897]: Waiting for error group managing segments... Aug 7 21:10:58.699426338 (4h44m12.286482579s) [upload=4897]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:10:58.699432380 (4h44m12.28648862s) [upload=4897]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:58.699436066 (4h44m12.286492317s) [upload=4897]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:10:58.775720472 (4h44m12.362776712s) [upload=4897]: Done uploading segments... Aug 7 21:10:58.775736111 (4h44m12.362792352s) [upload=4897]: Upload finished. err: <nil> Aug 7 21:11:31.613155564 (4h44m45.200211785s) [upload=4908]: Starting upload Aug 7 21:11:31.613232477 (4h44m45.200288698s) [upload=4908]: Done starting upload Aug 7 21:11:31.613307799 (4h44m45.20036401s) [upload=4908]: Uploading segments... Aug 7 21:11:31.613347051 (4h44m45.200403272s) [upload=4908]: Got next segment. Inline: true Aug 7 21:11:31.613352932 (4h44m45.200409153s) [upload=4908]: Waiting for error group managing segments... Aug 7 21:11:31.613404179 (4h44m45.2004604s) [upload=4908]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:11:31.613410591 (4h44m45.200466842s) [upload=4908]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:31.613414508 (4h44m45.200470729s) [upload=4908]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.682826735 (4h44m45.269882977s) [upload=4908]: Done uploading segments... Aug 7 21:11:31.682853265 (4h44m45.269909476s) [upload=4908]: Upload finished. err: <nil> Aug 7 21:11:51.754258397 (4h45m5.341314618s) [upload=4915]: Starting upload Aug 7 21:11:51.754330331 (4h45m5.341386542s) [upload=4915]: Done starting upload Aug 7 21:11:51.754383972 (4h45m5.341440193s) [upload=4915]: Uploading segments... Aug 7 21:11:51.754417083 (4h45m5.341473294s) [upload=4915]: Got next segment. Inline: true Aug 7 21:11:51.754420630 (4h45m5.34147684s) [upload=4915]: Waiting for error group managing segments... Aug 7 21:11:51.754453131 (4h45m5.341509352s) [upload=4915]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:11:51.754457108 (4h45m5.341513329s) [upload=4915]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:51.754460725 (4h45m5.341516936s) [upload=4915]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:11:51.830658438 (4h45m5.417714649s) [upload=4915]: Done uploading segments... Aug 7 21:11:51.830673094 (4h45m5.417729315s) [upload=4915]: Upload finished. err: <nil> Aug 7 21:12:54.778494646 (4h46m8.365550856s) [upload=4932]: Starting upload Aug 7 21:12:54.778560719 (4h46m8.36561693s) [upload=4932]: Done starting upload Aug 7 21:12:54.778608638 (4h46m8.365664859s) [upload=4932]: Uploading segments... Aug 7 21:12:54.778637322 (4h46m8.365693543s) [upload=4932]: Got next segment. Inline: true Aug 7 21:12:54.778640999 (4h46m8.36569722s) [upload=4932]: Waiting for error group managing segments... Aug 7 21:12:54.778677527 (4h46m8.365733739s) [upload=4932]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:54.778681655 (4h46m8.365737876s) [upload=4932]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:54.778684771 (4h46m8.365740983s) [upload=4932]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:12:54.868239378 (4h46m8.455295599s) [upload=4932]: Done uploading segments... Aug 7 21:12:54.868253865 (4h46m8.455310086s) [upload=4932]: Upload finished. err: <nil> Aug 7 21:13:23.238906839 (4h46m36.82596306s) [upload=4942]: Starting upload Aug 7 21:13:23.238959356 (4h46m36.826015567s) [upload=4942]: Done starting upload Aug 7 21:13:23.239011013 (4h46m36.826067234s) [upload=4942]: Uploading segments... Aug 7 21:13:23.239039727 (4h46m36.826095937s) [upload=4942]: Got next segment. Inline: true Aug 7 21:13:23.239043483 (4h46m36.826099695s) [upload=4942]: Waiting for error group managing segments... Aug 7 21:13:23.239098526 (4h46m36.826154737s) [upload=4942]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:13:23.239104707 (4h46m36.826160919s) [upload=4942]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:23.239108474 (4h46m36.826164696s) [upload=4942]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:23.309357825 (4h46m36.896414127s) [upload=4942]: Done uploading segments... Aug 7 21:13:23.309383974 (4h46m36.896440216s) [upload=4942]: Upload finished. err: <nil> Aug 7 21:14:18.294119361 (4h47m31.881175572s) [upload=4957]: Starting upload Aug 7 21:14:18.294187468 (4h47m31.881243679s) [upload=4957]: Done starting upload Aug 7 21:14:18.294261516 (4h47m31.881317737s) [upload=4957]: Uploading segments... Aug 7 21:14:18.294305068 (4h47m31.881361289s) [upload=4957]: Got next segment. Inline: true Aug 7 21:14:18.294311409 (4h47m31.881367621s) [upload=4957]: Waiting for error group managing segments... Aug 7 21:14:18.294349491 (4h47m31.881405712s) [upload=4957]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:18.294354059 (4h47m31.88141028s) [upload=4957]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:18.294356643 (4h47m31.881412854s) [upload=4957]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:15:18.280461255 (4h48m31.867517496s) [upload=4957]: Done uploading segments... Aug 7 21:15:18.280484308 (4h48m31.867540559s) [upload=4957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:18.387536247 (4h48m31.974592488s) [upload=4974]: Starting upload Aug 7 21:15:18.387607770 (4h48m31.974664011s) [upload=4974]: Done starting upload Aug 7 21:15:18.387666109 (4h48m31.974722361s) [upload=4974]: Uploading segments... Aug 7 21:15:18.387706996 (4h48m31.974763236s) [upload=4974]: Got next segment. Inline: true Aug 7 21:15:18.387712766 (4h48m31.974769007s) [upload=4974]: Waiting for error group managing segments... Aug 7 21:15:18.387755796 (4h48m31.974812038s) [upload=4974]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:18.387761227 (4h48m31.974817468s) [upload=4974]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:18.387764853 (4h48m31.974821095s) [upload=4974]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:18.373970870 (4h49m31.961027121s) [upload=4974]: Done uploading segments... Aug 7 21:16:18.374035110 (4h49m31.961091352s) [upload=4974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:18.454541239 (4h49m32.04159748s) [upload=4989]: Starting upload Aug 7 21:16:18.454621288 (4h49m32.041677529s) [upload=4989]: Done starting upload Aug 7 21:16:18.454699754 (4h49m32.041756005s) [upload=4989]: Uploading segments... Aug 7 21:16:18.454758153 (4h49m32.041814395s) [upload=4989]: Got next segment. Inline: true Aug 7 21:16:18.454765147 (4h49m32.041821398s) [upload=4989]: Waiting for error group managing segments... Aug 7 21:16:18.454813777 (4h49m32.041870018s) [upload=4989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:18.454819818 (4h49m32.04187606s) [upload=4989]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:18.454823425 (4h49m32.041879666s) [upload=4989]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:18.435319201 (4h50m32.022375413s) [upload=4989]: Done uploading segments... Aug 7 21:17:18.435335041 (4h50m32.022391262s) [upload=4989]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.000890115s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
30.000520003s
FAIL (2025-08-07 21:17:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000532366s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
1m0.000889122s
FAIL (2025-08-07 21:17:09 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.167316922 (4h46m37.754373133s) [noise=true , address=104.36.82.23:4457 , piece_id=3LDUSHS23Z6ZBTDHVGVNUZRFA3H2XDC6B243ISKKCYJJG3PCRUBA, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.167677125 (4h46m37.754733346s) [noise=true , address=104.36.82.37:4680 , piece_id=3LPKEUJ5HCZL4XY7YQOI2CROEHYNBFSZV6EUFDU2WV363FWQ644Q, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.167886165 (4h46m37.754942376s) [noise=true , address=64.49.13.27:4963 , piece_id=MN6MBMOOPEIPZRW5R4NQM6AEE3AHYX33MIXOSIIZEGHDPJEKAOPA, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.168145950 (4h46m37.755202161s) [noise=true , address=79.127.201.215:4354 , piece_id=4VVIMMVTX6SCVPFAODIC7JS5PVULFS2SCXMKFJKA72NXKEK62K3Q, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.168436171 (4h46m37.755492392s) [noise=true , address=104.36.82.13:4190 , piece_id=PVB3Y7KXU47SOQCZBKW4AUZ56XMLMJQ23WPM73BKHUMZKARYOCCA, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.168641956 (4h46m37.755698177s) [noise=true , address=45.91.116.10:4113 , piece_id=NJYFA7PI7F62J2CSEDJUTIWHY2LW3KBSLK76TKOVJNXC3RDV35KA, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.168845667 (4h46m37.755901877s) [noise=true , address=79.127.220.97:4396 , piece_id=HD3NDYSYACLTEX2OYRUDYNK5J2DREH3DTNSJSDO745Q6NCC7AXGA, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.169076598 (4h46m37.756132819s) [noise=true , address=104.36.82.44:4872 , piece_id=LM4MNB6MDHTN5MVJNGJOWX5Z5AKGICGK5JCLHBQBM5IR5PD3P2BQ, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.169509506 (4h46m37.756565727s) [noise=true , address=104.36.82.54:4183 , piece_id=PIBOQ56366UKNWMH43GSS6TW66GFPMSJOB3UQ3TKMEYEYPKVVFOQ, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.169710120 (4h46m37.756766341s) [noise=true , address=104.36.82.52:4108 , piece_id=AHQKVW323AG3TIIDCPX4HYFISVIY4V2476ZQM5GXLGLQD2PJ2KVA, seg_pos={0 0}, upload=4936]: Uploading piece...
Aug 7 21:13:24.260747004 (4h46m37.847803255s) [upload=4936]: Next returned nil segment
Aug 7 21:13:24.260784884 (4h46m37.847841125s) [upload=4936]: Waiting for error group managing segments...
Aug 7 21:13:24.282697478 (4h46m37.869753699s) [noise=true , address=104.36.82.36:4652 , piece_id=4S6GW6S5QMKXREUOZ4K7S77A5FIA5OGIS6MIPH3X5WFAIUYUTKBQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.286011234 (4h46m37.873067475s) [noise=true , address=45.91.116.10:4113 , piece_id=NJYFA7PI7F62J2CSEDJUTIWHY2LW3KBSLK76TKOVJNXC3RDV35KA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.288604064 (4h46m37.875660275s) [noise=true , address=104.36.82.23:4457 , piece_id=3LDUSHS23Z6ZBTDHVGVNUZRFA3H2XDC6B243ISKKCYJJG3PCRUBA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.289097404 (4h46m37.876153645s) [noise=true , address=104.36.82.44:4872 , piece_id=LM4MNB6MDHTN5MVJNGJOWX5Z5AKGICGK5JCLHBQBM5IR5PD3P2BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.303818073 (4h46m37.890874284s) [noise=true , address=104.36.82.20:4444 , piece_id=N3Z2NAQO5LHBPSABVMX4UBAAGUDDGDZCYGWX3NXWP6LUL5VJ74BA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.306609824 (4h46m37.893666035s) [noise=true , address=104.36.82.13:4190 , piece_id=PVB3Y7KXU47SOQCZBKW4AUZ56XMLMJQ23WPM73BKHUMZKARYOCCA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.307083188 (4h46m37.894139409s) [noise=true , address=104.36.82.46:4511 , piece_id=XTWLBKT2PX36GPDNTPPQ33HLJE2QMLNTVUMXVDEWJJO6IGC4HCXA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.333002740 (4h46m37.92005898s) [noise=true , address=104.36.82.11:4130 , piece_id=434EKDAN537AQFR342ZKLLENUIZAEK4PL2UTXDRJU7NJJ5HYHLUQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.335712287 (4h46m37.922768508s) [noise=true , address=79.127.220.97:4396 , piece_id=HD3NDYSYACLTEX2OYRUDYNK5J2DREH3DTNSJSDO745Q6NCC7AXGA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.364828819 (4h46m37.95188508s) [noise=true , address=104.36.82.54:4183 , piece_id=PIBOQ56366UKNWMH43GSS6TW66GFPMSJOB3UQ3TKMEYEYPKVVFOQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.415758456 (4h46m38.002814697s) [noise=true , address=104.36.82.37:4680 , piece_id=3LPKEUJ5HCZL4XY7YQOI2CROEHYNBFSZV6EUFDU2WV363FWQ644Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.427338251 (4h46m38.014394462s) [noise=true , address=104.36.82.43:4849 , piece_id=ZVVOBDRSIRLKPAOF23RMYRBXRNCNMQF3TJXYAW77ABOTGYUMXGHQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.490461549 (4h46m38.07751779s) [noise=true , address=79.127.201.215:4354 , piece_id=4VVIMMVTX6SCVPFAODIC7JS5PVULFS2SCXMKFJKA72NXKEK62K3Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.581523329 (4h46m38.16857954s) [noise=true , address=104.36.82.52:4108 , piece_id=AHQKVW323AG3TIIDCPX4HYFISVIY4V2476ZQM5GXLGLQD2PJ2KVA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil>
Aug 7 21:13:24.581552854 (4h46m38.168609065s) [seg_pos={0 0}, upload=4936]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:13:24.581774899 (4h46m38.16883112s) [noise=true , address=79.127.205.233:4643 , piece_id=IRDF6ZKJUCKCPSKAOCHW77H7O5ORLI43QULF2KWJ4O6SJTP373BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cVbdibJiX3T4fEUjeVBNNTTNaMb7m3QYDmxh17Dic5wVUQ9fx): protocol: expected piece hash; context canceled; EOF
Aug 7 21:13:24.581831154 (4h46m38.168887375s) [noise=true , address=104.36.82.62:4427 , piece_id=QMUVKOOD3THFKQAKBI3RQWH7JOP34MNNDOMKUXXBF4ZRVSYUJHJQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bx9fnLeXTPskzQB1Ad45qnq4CRivoGy7KXALY35dzVGF6S5Ep): protocol: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection
Aug 7 21:13:24.581866029 (4h46m38.168922239s) [noise=true , address=64.49.13.154:4933 , piece_id=HGHDC3TAI5HTPACRHKMJMLCMMF2AZOMT4DO5LIQEIECYYKCHMIRA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n5Mn8FNLXxFWtbbaMv8VuyxuXe6PiiMVtLETzm5aKfxhEykt3): protocol: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection
Aug 7 21:13:24.581904450 (4h46m38.168960661s) [noise=true , address=64.49.13.27:4963 , piece_id=MN6MBMOOPEIPZRW5R4NQM6AEE3AHYX33MIXOSIIZEGHDPJEKAOPA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq): protocol: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection
Aug 7 21:13:24.581925911 (4h46m38.168982132s) [seg_pos={0 0}, upload=4936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:13:24.582013845 (4h46m38.169070066s) [upload=4936]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:13:24.582018403 (4h46m38.169074624s) [upload=4936]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:15:00.743765287 (4h48m14.330821508s) [upload=4936]: Done uploading segments...
Aug 7 21:15:00.743795124 (4h48m14.330851345s) [upload=4936]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:15:00.812344761 (4h48m14.399400973s) [upload=4967]: Starting upload
Aug 7 21:15:00.812384405 (4h48m14.399440606s) [upload=4967]: Done starting upload
Aug 7 21:15:00.812509518 (4h48m14.399565729s) [upload=4967]: Uploading segments...
Aug 7 21:15:00.812592684 (4h48m14.399648895s) [upload=4967]: Got next segment. Inline: false
Aug 7 21:15:00.812634802 (4h48m14.399691023s) [upload=4967]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:15:00.812639241 (4h48m14.399695462s) [upload=4967]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:16:00.800009657 (4h49m14.387065878s) [upload=4967]: Next segment err: context deadline exceeded
Aug 7 21:16:00.800074909 (4h49m14.38713113s) [upload=4967]: Done uploading segments...
Aug 7 21:16:00.800088334 (4h49m14.387144545s) [upload=4967]: Upload finished. err: context deadline exceeded
Aug 7 21:16:09.662919397 (4h49m23.249975618s) [upload=4987]: Starting upload
Aug 7 21:16:09.663003895 (4h49m23.250060116s) [upload=4987]: Done starting upload
Aug 7 21:16:09.663197185 (4h49m23.250253426s) [upload=4987]: Uploading segments...
Aug 7 21:16:09.663304947 (4h49m23.250361248s) [upload=4987]: Got next segment. Inline: false
Aug 7 21:16:09.663375208 (4h49m23.250431459s) [upload=4987]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:16:09.663382832 (4h49m23.250439074s) [upload=4987]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:17:09.645661646 (4h50m23.232717887s) [upload=4987]: Next segment err: context deadline exceeded
Aug 7 21:17:09.645901063 (4h50m23.232957304s) [upload=4987]: Done uploading segments...
Aug 7 21:17:09.645976574 (4h50m23.233032815s) [upload=4987]: Upload finished. err: context deadline exceeded
Duration: 1m0.000918728s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
701.63245ms
FAIL (2025-08-07 21:17:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: too many requests
======LOGS======
L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece...
Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment
Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments...
Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil>
Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF
Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection
Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection
Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments...
Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil>
Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload
Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload
Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments...
Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false
Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests
Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments...
Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests
Duration: 701.654473ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
7 out of 60 is failed (threshold=3) [...................................................XX..XXXXX] duration:
30.000303028s
FAIL (2025-08-07 21:17:03 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000318597s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.......................................................XX.XX] duration:
1m0.000607794s
FAIL (2025-08-07 21:16:51 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000619776s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration:
30.000918875s
FAIL (2025-08-07 21:16:50 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7s) [upload=4861]: Done uploading segments... Aug 7 21:08:51.240335075 (4h42m4.827391316s) [upload=4861]: Upload finished. err: <nil> Aug 7 21:09:58.632781085 (4h43m12.219837326s) [upload=4880]: Starting upload Aug 7 21:09:58.632867045 (4h43m12.219923286s) [upload=4880]: Done starting upload Aug 7 21:09:58.632946213 (4h43m12.220002465s) [upload=4880]: Uploading segments... Aug 7 21:09:58.632990476 (4h43m12.220046716s) [upload=4880]: Got next segment. Inline: true Aug 7 21:09:58.632996186 (4h43m12.220052437s) [upload=4880]: Waiting for error group managing segments... Aug 7 21:09:58.633044917 (4h43m12.220101159s) [upload=4880]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:09:58.633050438 (4h43m12.220106679s) [upload=4880]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:09:58.633077508 (4h43m12.220133759s) [upload=4880]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:09:58.687844850 (4h43m12.274901172s) [upload=4880]: Done uploading segments... Aug 7 21:09:58.687873194 (4h43m12.274929435s) [upload=4880]: Upload finished. err: <nil> Aug 7 21:10:48.980309603 (4h44m2.567365814s) [upload=4895]: Starting upload Aug 7 21:10:48.980351321 (4h44m2.567407542s) [upload=4895]: Done starting upload Aug 7 21:10:48.980412926 (4h44m2.567469137s) [upload=4895]: Uploading segments... Aug 7 21:10:48.980468068 (4h44m2.567524309s) [upload=4895]: Got next segment. Inline: true Aug 7 21:10:48.980474410 (4h44m2.567530621s) [upload=4895]: Waiting for error group managing segments... Aug 7 21:10:48.980519745 (4h44m2.567575966s) [upload=4895]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:10:48.980525596 (4h44m2.567581807s) [upload=4895]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:48.980529363 (4h44m2.567585584s) [upload=4895]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:10:49.036078074 (4h44m2.623134295s) [upload=4895]: Done uploading segments... Aug 7 21:10:49.036092611 (4h44m2.623148821s) [upload=4895]: Upload finished. err: <nil> Aug 7 21:11:10.299224768 (4h44m23.886281009s) [upload=4903]: Starting upload Aug 7 21:11:10.299304937 (4h44m23.886361178s) [upload=4903]: Done starting upload Aug 7 21:11:10.299368445 (4h44m23.886424686s) [upload=4903]: Uploading segments... Aug 7 21:11:10.299409052 (4h44m23.886465292s) [upload=4903]: Got next segment. Inline: true Aug 7 21:11:10.299414652 (4h44m23.886470893s) [upload=4903]: Waiting for error group managing segments... Aug 7 21:11:10.299460468 (4h44m23.886516709s) [upload=4903]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:11:10.299465988 (4h44m23.886522229s) [upload=4903]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:10.299469444 (4h44m23.886525685s) [upload=4903]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:11:10.444467755 (4h44m24.031523996s) [upload=4903]: Done uploading segments... Aug 7 21:11:10.444485819 (4h44m24.03154206s) [upload=4903]: Upload finished. err: <nil> Aug 7 21:12:08.543001654 (4h45m22.130057896s) [upload=4919]: Starting upload Aug 7 21:12:08.543123191 (4h45m22.130179432s) [upload=4919]: Done starting upload Aug 7 21:12:08.543216246 (4h45m22.130272487s) [upload=4919]: Uploading segments... Aug 7 21:12:08.543261810 (4h45m22.130318051s) [upload=4919]: Got next segment. Inline: true Aug 7 21:12:08.543267651 (4h45m22.130323892s) [upload=4919]: Waiting for error group managing segments... Aug 7 21:12:08.543313246 (4h45m22.130369488s) [upload=4919]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:08.543319278 (4h45m22.130375519s) [upload=4919]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:08.543323075 (4h45m22.130379316s) [upload=4919]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:12:08.630954371 (4h45m22.218010612s) [upload=4919]: Done uploading segments... Aug 7 21:12:08.630979197 (4h45m22.218035449s) [upload=4919]: Upload finished. err: <nil> Aug 7 21:12:30.034507380 (4h45m43.62156362s) [upload=4926]: Starting upload Aug 7 21:12:30.034584302 (4h45m43.621640544s) [upload=4926]: Done starting upload Aug 7 21:12:30.034643523 (4h45m43.621699754s) [upload=4926]: Uploading segments... Aug 7 21:12:30.034681965 (4h45m43.621738206s) [upload=4926]: Got next segment. Inline: true Aug 7 21:12:30.034687645 (4h45m43.621743886s) [upload=4926]: Waiting for error group managing segments... Aug 7 21:12:30.034730766 (4h45m43.621787007s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:30.034735976 (4h45m43.621792207s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:30.034739642 (4h45m43.621795883s) [upload=4926]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:00.022476261 (4h46m13.609532503s) [upload=4926]: Done uploading segments... Aug 7 21:13:00.022497211 (4h46m13.609553452s) [upload=4926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:15.702096720 (4h46m29.289152941s) [upload=4939]: Starting upload Aug 7 21:13:15.702154518 (4h46m29.289210739s) [upload=4939]: Done starting upload Aug 7 21:13:15.702196716 (4h46m29.289252928s) [upload=4939]: Uploading segments... Aug 7 21:13:15.702224368 (4h46m29.289280589s) [upload=4939]: Got next segment. Inline: true Aug 7 21:13:15.702228446 (4h46m29.289284657s) [upload=4939]: Waiting for error group managing segments... Aug 7 21:13:15.702261458 (4h46m29.289317669s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:13:15.702265306 (4h46m29.289321527s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:15.702267880 (4h46m29.289324101s) [upload=4939]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.689162551 (4h46m59.276218792s) [upload=4939]: Done uploading segments... Aug 7 21:13:45.689208046 (4h46m59.276264277s) [upload=4939]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:13.522007492 (4h47m27.109063713s) [upload=4955]: Starting upload Aug 7 21:14:13.522102209 (4h47m27.10915843s) [upload=4955]: Done starting upload Aug 7 21:14:13.522154938 (4h47m27.109211179s) [upload=4955]: Uploading segments... Aug 7 21:14:13.522235488 (4h47m27.109291729s) [upload=4955]: Got next segment. Inline: true Aug 7 21:14:13.522242682 (4h47m27.109298922s) [upload=4955]: Waiting for error group managing segments... Aug 7 21:14:13.522293467 (4h47m27.109349708s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:13.522299518 (4h47m27.109355759s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:13.522303275 (4h47m27.109359516s) [upload=4955]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:14:43.507883903 (4h47m57.094940144s) [upload=4955]: Done uploading segments... Aug 7 21:14:43.507905914 (4h47m57.094962156s) [upload=4955]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:54.825501522 (4h48m8.412557734s) [upload=4964]: Starting upload Aug 7 21:14:54.825580730 (4h48m8.412636951s) [upload=4964]: Done starting upload Aug 7 21:14:54.825629411 (4h48m8.412685632s) [upload=4964]: Uploading segments... Aug 7 21:14:54.825660400 (4h48m8.41271661s) [upload=4964]: Got next segment. Inline: true Aug 7 21:14:54.825664547 (4h48m8.412720758s) [upload=4964]: Waiting for error group managing segments... Aug 7 21:14:54.825705873 (4h48m8.412762115s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:54.825711855 (4h48m8.412768076s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:54.825715662 (4h48m8.412771883s) [upload=4964]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:15:24.812961328 (4h48m38.400017569s) [upload=4964]: Done uploading segments... Aug 7 21:15:24.812988770 (4h48m38.40004501s) [upload=4964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:40.993035983 (4h48m54.580092194s) [upload=4978]: Starting upload Aug 7 21:15:40.993105403 (4h48m54.580161614s) [upload=4978]: Done starting upload Aug 7 21:15:40.993154163 (4h48m54.580210375s) [upload=4978]: Uploading segments... Aug 7 21:15:40.993189519 (4h48m54.580245741s) [upload=4978]: Got next segment. Inline: true Aug 7 21:15:40.993194319 (4h48m54.58025054s) [upload=4978]: Waiting for error group managing segments... Aug 7 21:15:40.993226930 (4h48m54.580283151s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:40.993231167 (4h48m54.580287379s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:40.993234264 (4h48m54.580290474s) [upload=4978]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:10.982081529 (4h49m24.569137771s) [upload=4978]: Done uploading segments... Aug 7 21:16:10.982104473 (4h49m24.569160713s) [upload=4978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:20.227399180 (4h49m33.814455401s) [upload=4990]: Starting upload Aug 7 21:16:20.227450334 (4h49m33.814506545s) [upload=4990]: Done starting upload Aug 7 21:16:20.227491060 (4h49m33.814547272s) [upload=4990]: Uploading segments... Aug 7 21:16:20.227516609 (4h49m33.81457283s) [upload=4990]: Got next segment. Inline: true Aug 7 21:16:20.227520536 (4h49m33.814576757s) [upload=4990]: Waiting for error group managing segments... Aug 7 21:16:20.227552426 (4h49m33.814608636s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:20.227556263 (4h49m33.814612484s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:20.227558527 (4h49m33.814614739s) [upload=4990]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:50.216081431 (4h50m3.803137652s) [upload=4990]: Done uploading segments... Aug 7 21:16:50.216097492 (4h50m3.803153713s) [upload=4990]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.000934935s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
30.002384376s
FAIL (2025-08-07 21:16:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.002403052s
edge
gateway
us-select-1
|