Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
24.932147ms remote:
"tcp:136.0.77.3:443" response headers:
Content-Security-Policy=block-all-mixed-content Content-Type=application/xml Vary=Origin Content-Length=274 Server=Storj X-Amz-Request-Id=18974366DB00E5B0 X-Request-Id=7CMwhGyvbCm X-Xss-Protection=1; mode=block Date=Tue, 24 Feb 2026 18:51:27 GMT Accept-Ranges=bytes target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011 trace:
https://jaeger.dev.storj.io/trace/7f526b7a33eee0ff
FAIL (2026-02-24 18:51:27 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>18974366DB00E5B0</RequestId><HostId></HostId></Error>"
Duration: 24.95516ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
40 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XX.......] duration:
1.700549495s trace:
https://jaeger.dev.storj.io/trace/c09458d54de96a9
FAIL (2026-02-24 18:51:26 UTC)
operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 1.70056853s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
29 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX............] duration:
503.686245ms trace:
https://jaeger.dev.storj.io/trace/2d0954cf9daf2bbb
FAIL (2026-02-24 18:51:26 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Feb 24 18:42:00.490044390 (166h33m19.629028604s) [upload=146475]: Starting upload Feb 24 18:42:00.490207464 (166h33m19.629191679s) [upload=146475]: Done starting upload Feb 24 18:42:00.490279810 (166h33m19.629264024s) [upload=146475]: Uploading segments... Feb 24 18:42:00.490360300 (166h33m19.629344514s) [upload=146475]: Got next segment. Inline: true Feb 24 18:42:00.490367984 (166h33m19.629352198s) [upload=146475]: Waiting for error group managing segments... Feb 24 18:42:00.490436152 (166h33m19.629420366s) [upload=146475]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Feb 24 18:42:00.490442564 (166h33m19.629426788s) [upload=146475]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Feb 24 18:42:00.490447063 (166h33m19.629431276s) [upload=146475]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Feb 24 18:43:00.472138274 (166h34m19.611122487s) [upload=146475]: Done uploading segments... Feb 24 18:43:00.472188278 (166h34m19.611172491s) [upload=146475]: Upload finished. err: metaclient: context deadline exceeded
Duration: 503.705111ms
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
25.546155ms remote:
"tcp:136.0.77.3:443" response headers:
X-Xss-Protection=1; mode=block Content-Length=274 Date=Tue, 24 Feb 2026 18:51:20 GMT Accept-Ranges=bytes Content-Security-Policy=block-all-mixed-content Content-Type=application/xml Server=Storj Vary=Origin X-Amz-Request-Id=189743653622A797 X-Request-Id=ZaotmLpqQs7 target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/7f140c35fb9091ac
FAIL (2026-02-24 18:51:20 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>189743653622A797</RequestId><HostId></HostId></Error>"
Duration: 25.563818ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
43 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.........] duration:
189.505868ms trace:
https://jaeger.dev.storj.io/trace/5383e9cc16a1cf07
FAIL (2026-02-24 18:51:18 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 189.527539ms
core
uplink
us1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
13.285167ms remote:
"tcp:45.38.24.2:443" response headers:
Content-Length=284 Content-Type=application/xml X-Amz-Request-Id=1897436352B4C8D8 X-Request-Id=HBoCLaXnckV X-Xss-Protection=1; mode=block Content-Security-Policy=block-all-mixed-content Server=Storj Vary=Origin Date=Tue, 24 Feb 2026 18:51:12 GMT Accept-Ranges=bytes target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/3c50e9fad1b45ab6
FAIL (2026-02-24 18:51:12 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897436352B4C8D8</RequestId><HostId></HostId></Error>"
Duration: 13.306739ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
30 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........] duration:
580.481941ms trace:
https://jaeger.dev.storj.io/trace/51430d55a27ab7ca
FAIL (2026-02-24 18:51:09 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 580.501758ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
41 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.........] duration:
219.030654ms trace:
https://jaeger.dev.storj.io/trace/5b29a0cacadf86f0
FAIL (2026-02-24 18:51:08 UTC)
operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Feb 24 18:44:31.418187254 (166h35m50.557171448s) [upload=146507]: Starting upload Feb 24 18:44:31.418228731 (166h35m50.557212916s) [upload=146507]: Done starting upload Feb 24 18:44:31.418274858 (166h35m50.557259042s) [upload=146507]: Uploading segments... Feb 24 18:44:31.418305145 (166h35m50.557289339s) [upload=146507]: Got next segment. Inline: true Feb 24 18:44:31.418308962 (166h35m50.557293156s) [upload=146507]: Waiting for error group managing segments... Feb 24 18:44:31.418363463 (166h35m50.557347657s) [upload=146507]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Feb 24 18:44:31.418369304 (166h35m50.557353488s) [upload=146507]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Feb 24 18:44:31.418372259 (166h35m50.557356443s) [upload=146507]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Feb 24 18:44:50.941040072 (166h36m10.080024286s) [upload=146507]: Done uploading segments... Feb 24 18:44:50.941058296 (166h36m10.08004251s) [upload=146507]: Upload finished. err: <nil>
Duration: 219.054509ms
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
15 out of 60 is failed (threshold=6) [................................XXX.XXXXXX.XXXXXX...........] duration:
133.957774ms remote:
"tcp:136.0.77.4:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/6c3b9ac24855cb84
FAIL (2026-02-24 18:51:08 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 133.982ms
edge
linksharing
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
28.905721ms remote:
"tcp:136.0.77.3:443" response headers:
Content-Length=274 Content-Security-Policy=block-all-mixed-content Content-Type=application/xml X-Amz-Request-Id=1897436111F093C5 X-Xss-Protection=1; mode=block Date=Tue, 24 Feb 2026 18:51:02 GMT Accept-Ranges=bytes Server=Storj Vary=Origin X-Request-Id=FYb1gJTsEac target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/1033cb7579f224f5
FAIL (2026-02-24 18:51:02 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897436111F093C5</RequestId><HostId></HostId></Error>"
Duration: 28.920107ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
42 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........] duration:
795.938349ms trace:
https://jaeger.dev.storj.io/trace/6973a18e73267103
FAIL (2026-02-24 18:51:01 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 795.961924ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [........X.....................X......X.XXX...X.....X........] duration:
1.013992249s trace:
https://jaeger.dev.storj.io/trace/29c8bea8f1a363cb
FAIL (2026-02-24 18:50:58 UTC)
operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.014017395s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
44 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.......] duration:
907.373346ms trace:
https://jaeger.dev.storj.io/trace/4c427ad68ce4a878
FAIL (2026-02-24 18:50:53 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 907.433548ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
29 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX..........] duration:
849.24282ms trace:
https://jaeger.dev.storj.io/trace/7ea78b408ceac77f
FAIL (2026-02-24 18:50:42 UTC)
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1s) [noise=true , address=79.127.219.41:4948 , piece_id=AKMCPMBJICKLCUO5EDO2VTOGHO3EOBF2ZGUZOVE2JBAJZ7AL4GKQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.919349510 (166h35m20.058333704s) [noise=true , address=79.127.219.46:4900 , piece_id=E6Z43XUEXNCGI76F2Z7WDCK6E5CZMMSBN4NE3VQKT3HAFS2Y7QSQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.919385177 (166h35m20.058369371s) [noise=true , address=45.91.116.13:4179 , piece_id=OQ624U2HU36XD77IKJBP6Z6B3QHEOWW3MLQLOHDTK34N7C25A3FA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.919848713 (166h35m20.058832897s) [noise=true , address=77.90.31.19:4855 , piece_id=ZF7VMBTCLI2YMUMUL3DNJ2UPVIN5WNWM5CVR56T6F3P6LMAAZZCQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920149715 (166h35m20.059133899s) [noise=true , address=45.91.116.6:4985 , piece_id=GGC7CI62BZZB56QJKHL7MZVPBZCNJKGG3XN5JYUD3WB3C3G67XIA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920391487 (166h35m20.059375681s) [noise=true , address=45.91.116.12:4252 , piece_id=ZZ4SDZDF5B7KGQXX2CR6KVKSCCJGWYI327TDMKVEV6VVJNVHFVGA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920646944 (166h35m20.059631128s) [noise=true , address=77.90.31.21:4911 , piece_id=PAC75B4GQR7ZGOONLLG3QZHM62WBGPQGUDJ3QSOHUHBQSGWAVC5Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920909065 (166h35m20.059893248s) [noise=true , address=145.79.227.35:4491 , piece_id=QRWSMM5V6XF75ORBT5E4EONQHZ3BSRKTRJKF5R2OYM7RHWUYOETQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921156867 (166h35m20.060141051s) [noise=true , address=77.90.31.13:4636 , piece_id=UQ5MUJ3MVVLEGB7GH5BXC6XWZY7U2OKSXX3WFOD6OHZYFPOBNKJQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921400382 (166h35m20.060384576s) [noise=true , address=193.149.164.66:4972 , piece_id=22NWJODMK6GYWQQRUGQDTR7AMOBFXSNK5VE26N75OKEY36YS2CVA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921659165 (166h35m20.060643349s) [noise=true , address=45.91.116.8:4052 , piece_id=OSFGARHZ4RC3GB6TGL57OUS6G5HML7HBLEPVMQQM3HWJO6UBTA7Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921973843 (166h35m20.060958037s) [noise=true , address=77.90.31.16:4742 , piece_id=UQLVEP6CSK7AQNGD63UP2CPV7AIA2NIZ4RZXCPOZXJPOBRM7TH6Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.922281628 (166h35m20.061265822s) [noise=true , address=145.79.227.34:4939 , piece_id=KFYQXKIMP5FUYCXO5ADS322HOPEBT33QZ7ZBIYXLUGEXMVVEXLMA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.922631973 (166h35m20.061616157s) [noise=true , address=145.79.227.37:4582 , piece_id=3KJKXV6QUMY2WXQQ65MF6PR3IKXQ6BGHPBOCCRMBG3BPGCSQIXEQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.924956486 (166h35m20.06394068s) [noise=true , address=145.79.227.14:4846 , piece_id=QXT2YVIRCQT7HKZKTJR4BRQRNTWGAFYVQF4YZEIKCRA5YY77ZLJA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.926527841 (166h35m20.065512036s) [noise=true , address=145.79.227.29:4970 , piece_id=KENFISHR4HYVPMBMD33G2E5G6LARJZN3NQYOZ4KKJ7ODMLNZQT2Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.928105218 (166h35m20.067089402s) [noise=true , address=45.91.116.14:4204 , piece_id=XQQZ4DIUCLHLQILPOYN2RL7ON5VRJCFS622OJC6Y4Y7RSG6B4W7Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.929764789 (166h35m20.068748983s) [noise=true , address=77.90.31.14:4689 , piece_id=FS3QNBO7K3N3ZIZYPV57AAIMJQ33NKCGT4GBCA7THC5QDEO7YEQQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:01.012045757 (166h35m20.151029971s) [upload=146493]: Next returned nil segment
Feb 24 18:44:01.012057469 (166h35m20.151041683s) [upload=146493]: Waiting for error group managing segments...
Feb 24 18:44:01.020560936 (166h35m20.15954515s) [noise=true , address=45.91.116.8:4052 , piece_id=OSFGARHZ4RC3GB6TGL57OUS6G5HML7HBLEPVMQQM3HWJO6UBTA7Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.022333197 (166h35m20.161317401s) [noise=true , address=79.127.219.46:4900 , piece_id=E6Z43XUEXNCGI76F2Z7WDCK6E5CZMMSBN4NE3VQKT3HAFS2Y7QSQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.035953132 (166h35m20.174937356s) [noise=true , address=145.79.227.35:4491 , piece_id=QRWSMM5V6XF75ORBT5E4EONQHZ3BSRKTRJKF5R2OYM7RHWUYOETQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.036725164 (166h35m20.175709388s) [noise=true , address=45.91.116.12:4252 , piece_id=ZZ4SDZDF5B7KGQXX2CR6KVKSCCJGWYI327TDMKVEV6VVJNVHFVGA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.041227315 (166h35m20.180211529s) [noise=true , address=45.91.116.6:4985 , piece_id=GGC7CI62BZZB56QJKHL7MZVPBZCNJKGG3XN5JYUD3WB3C3G67XIA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.044901219 (166h35m20.183885433s) [noise=true , address=45.91.116.14:4204 , piece_id=XQQZ4DIUCLHLQILPOYN2RL7ON5VRJCFS622OJC6Y4Y7RSG6B4W7Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.046613308 (166h35m20.185597521s) [noise=true , address=77.90.31.16:4742 , piece_id=UQLVEP6CSK7AQNGD63UP2CPV7AIA2NIZ4RZXCPOZXJPOBRM7TH6Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.049905900 (166h35m20.188890123s) [noise=true , address=77.90.31.19:4855 , piece_id=ZF7VMBTCLI2YMUMUL3DNJ2UPVIN5WNWM5CVR56T6F3P6LMAAZZCQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.050217330 (166h35m20.189201544s) [noise=true , address=77.90.31.13:4636 , piece_id=UQ5MUJ3MVVLEGB7GH5BXC6XWZY7U2OKSXX3WFOD6OHZYFPOBNKJQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.063062278 (166h35m20.202046492s) [noise=true , address=77.90.31.14:4689 , piece_id=FS3QNBO7K3N3ZIZYPV57AAIMJQ33NKCGT4GBCA7THC5QDEO7YEQQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.063877701 (166h35m20.202861915s) [noise=true , address=45.91.116.13:4179 , piece_id=OQ624U2HU36XD77IKJBP6Z6B3QHEOWW3MLQLOHDTK34N7C25A3FA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.066973164 (166h35m20.205957379s) [noise=true , address=77.90.31.21:4911 , piece_id=PAC75B4GQR7ZGOONLLG3QZHM62WBGPQGUDJ3QSOHUHBQSGWAVC5Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.093746163 (166h35m20.232730377s) [noise=true , address=145.79.227.14:4846 , piece_id=QXT2YVIRCQT7HKZKTJR4BRQRNTWGAFYVQF4YZEIKCRA5YY77ZLJA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.099012601 (166h35m20.237996825s) [noise=true , address=145.79.227.29:4970 , piece_id=KENFISHR4HYVPMBMD33G2E5G6LARJZN3NQYOZ4KKJ7ODMLNZQT2Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.099029282 (166h35m20.238013496s) [seg_pos={0 0}, upload=146493]: Satisfied cohort requirements after 14 pieces
Feb 24 18:44:01.099459185 (166h35m20.2384434s) [noise=true , address=79.127.219.41:4948 , piece_id=AKMCPMBJICKLCUO5EDO2VTOGHO3EOBF2ZGUZOVE2JBAJZ7AL4GKQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gXNjKMdu97i8BC5fYWMiMf3FTepsmHsViuksBWNVLpraTJ1XW): protocol: expected piece hash; context canceled; EOF
Feb 24 18:44:01.099506874 (166h35m20.238491089s) [noise=true , address=79.127.219.41:4948 , piece_id=AKMCPMBJICKLCUO5EDO2VTOGHO3EOBF2ZGUZOVE2JBAJZ7AL4GKQ, seg_pos={0 0}, upload=146493]: Long tail cancellation due to optimal threshold
Feb 24 18:44:01.099675470 (166h35m20.238659684s) [noise=true , address=145.79.227.37:4582 , piece_id=3KJKXV6QUMY2WXQQ65MF6PR3IKXQ6BGHPBOCCRMBG3BPGCSQIXEQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn): protocol: expected piece hash; context canceled; EOF
Feb 24 18:44:01.099683645 (166h35m20.238667859s) [noise=true , address=145.79.227.37:4582 , piece_id=3KJKXV6QUMY2WXQQ65MF6PR3IKXQ6BGHPBOCCRMBG3BPGCSQIXEQ, seg_pos={0 0}, upload=146493]: Long tail cancellation due to optimal threshold
Feb 24 18:44:01.099865534 (166h35m20.238849749s) [noise=true , address=193.149.164.66:4972 , piece_id=22NWJODMK6GYWQQRUGQDTR7AMOBFXSNK5VE26N75OKEY36YS2CVA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128eDneuvfeZbqFp8YQhWHLb7hWrw7T3a2EJ7tzZkkYhaNq7CMT): protocol: read tcp 5.161.89.252:24006->193.149.164.66:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 24 18:44:01.099872908 (166h35m20.238857132s) [noise=true , address=193.149.164.66:4972 , piece_id=22NWJODMK6GYWQQRUGQDTR7AMOBFXSNK5VE26N75OKEY36YS2CVA, seg_pos={0 0}, upload=146493]: Long tail cancellation due to optimal threshold
Feb 24 18:44:01.100082470 (166h35m20.239066684s) [noise=true , address=145.79.227.34:4939 , piece_id=KFYQXKIMP5FUYCXO5ADS322HOPEBT33QZ7ZBIYXLUGEXMVVEXLMA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.100105593 (166h35m20.239089808s) [seg_pos={0 0}, upload=146493]: Satisfied cohort requirements after 15 pieces
Feb 24 18:44:01.100170554 (166h35m20.239154778s) [seg_pos={0 0}, upload=146493]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Feb 24 18:44:01.100364717 (166h35m20.239348941s) [upload=146493]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 24 18:44:01.100379766 (166h35m20.23936399s) [upload=146493]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 24 18:44:14.739172222 (166h35m33.878156437s) [upload=146493]: Done uploading segments...
Feb 24 18:44:14.739199784 (166h35m33.878184008s) [upload=146493]: Upload finished. err: <nil>
Duration: 849.25887ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
40 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XX......] duration:
1.761497736s trace:
https://jaeger.dev.storj.io/trace/c09458d54de96a9
FAIL (2026-02-24 18:50:38 UTC)
operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 1.761524917s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
43 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X........] duration:
151.121279ms trace:
https://jaeger.dev.storj.io/trace/5383e9cc16a1cf07
FAIL (2026-02-24 18:50:37 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 151.140545ms
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXX.........] duration:
395.614311ms trace:
https://jaeger.dev.storj.io/trace/7b154e70da6a7540
FAIL (2026-02-24 18:50:35 UTC)
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Feb 24 18:41:57.999131463 (166h33m17.138115677s) [upload=146474]: Starting upload
Feb 24 18:41:57.999198959 (166h33m17.138183174s) [upload=146474]: Done starting upload
Feb 24 18:41:57.999328141 (166h33m17.138312335s) [upload=146474]: Uploading segments...
Feb 24 18:41:57.999615107 (166h33m17.138599331s) [upload=146474]: Got next segment. Inline: false
Feb 24 18:41:57.999672624 (166h33m17.138656837s) [upload=146474]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 24 18:41:57.999678475 (166h33m17.138662688s) [upload=146474]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 24 18:42:09.429844419 (166h33m28.568828603s) [seg_pos={0 0}, upload=146474]: Begin upload segment...
Feb 24 18:42:09.429878984 (166h33m28.568863178s) [seg_pos={0 0}, upload=146474]: Done begin upload segment.
Feb 24 18:42:09.429901105 (166h33m28.568885299s) [noise=true , address=145.79.227.11:4731 , piece_id=MX5CYRWIYNI55NDKVMWGA3AFS2LMUPUU34246CNIBA24BWDVARIQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.429935218 (166h33m28.568919403s) [noise=true , address=145.79.227.35:4486 , piece_id=D6SHML7OT3YNTWYLFOQWBDLUZCBOE3HIU4E7N2F7SLGOGCUQ5RMQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.430230660 (166h33m28.569214854s) [noise=true , address=145.79.227.15:4875 , piece_id=FXYLYOG4Y2MQ5NKOYUVPKZU7NOIECZGBAQT5B5WTYGHJ7TQ54C4A, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.430492840 (166h33m28.569477024s) [noise=true , address=145.79.227.43:4157 , piece_id=WM3UGYNXPAYIWMZT3DXFLV44LWGMAKHN7YUWG4A2HOO26HSPCGKQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.430750783 (166h33m28.569734967s) [noise=true , address=145.79.227.16:4914 , piece_id=LOH6G33NOTSPX2EHGYPSEXHCR2VLUWRR2BYRZO5KATQSN3KUTTZA, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.430991071 (166h33m28.569975256s) [noise=true , address=145.79.227.49:4343 , piece_id=HJMNJUJ62JDEKJH55MG4ARTKENXN6ENSAB6SO2VXXCXF7CSGS6PQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.431233163 (166h33m28.570217357s) [noise=true , address=45.91.116.5:4297 , piece_id=WZJLLLQIUTPKW247TATRLH5K425MX2ZD76JNGJQT4KME7AHV2FFA, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.431551629 (166h33m28.570535813s) [noise=true , address=145.79.227.37:4567 , piece_id=KLDMATQWJ5BJBN6EH76Q27YKWK6MFT74AAR543RYW2NX6YBHCO7Q, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.431839126 (166h33m28.57082332s) [noise=true , address=145.79.227.20:4443 , piece_id=MYRORKSVAKD76XH6EJ7NVP5P5YNQ7TLESLF2LAA22B6PH5SHTQCQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.433266051 (166h33m28.572250245s) [noise=true , address=145.79.227.41:4088 , piece_id=S25UWWCAVJUXYEGMU45PHVTQX4L5BRFAKEIHDQD426UYI5XOEZIA, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.434446958 (166h33m28.573431142s) [noise=true , address=145.79.227.39:4018 , piece_id=3QEOPGVK32JLEJV2P4HVZY763WAZCMBKFKRKYKAKW2RUZIAUXODQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.435448058 (166h33m28.574432252s) [noise=true , address=77.90.31.18:4805 , piece_id=6VL6GAR26VPUFXNMERHUYB746PA72NZDA7LTJU3CHSRGV6EYPWDQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.436699445 (166h33m28.575683639s) [noise=true , address=145.79.227.38:4978 , piece_id=DZ3YISF75444FUUJGNDVG3SPZSWOIRJEVZYAKTCG73D7KIYXOXRQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.438093540 (166h33m28.577077734s) [noise=true , address=145.79.227.26:4653 , piece_id=NR3KMDIWN4WZM7ASUFHNBJJZPOI5IZJJLWTEZO6IYMRYMA4IO2TQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.439307297 (166h33m28.578291481s) [noise=true , address=145.79.227.2:4114 , piece_id=RRA45YDH5ZDU4CH5CTNQMKWPFENHYDEGE7V3NKLJ3LRXGFQORL5Q, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.439761746 (166h33m28.57874594s) [noise=true , address=145.79.227.40:4049 , piece_id=NPRJTV5UJRTE52IDTVHMFXJD7CYR2I5C2ZFLUJVAVD6ZUZMVZNAQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.518930921 (166h33m28.657915115s) [upload=146474]: Next returned nil segment
Feb 24 18:42:09.518940428 (166h33m28.657924612s) [upload=146474]: Waiting for error group managing segments...
Feb 24 18:42:09.528008379 (166h33m28.666992563s) [noise=true , address=145.79.227.15:4875 , piece_id=FXYLYOG4Y2MQ5NKOYUVPKZU7NOIECZGBAQT5B5WTYGHJ7TQ54C4A, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.528221628 (166h33m28.667205812s) [noise=true , address=145.79.227.39:4018 , piece_id=3QEOPGVK32JLEJV2P4HVZY763WAZCMBKFKRKYKAKW2RUZIAUXODQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.528377569 (166h33m28.667361753s) [noise=true , address=145.79.227.49:4343 , piece_id=HJMNJUJ62JDEKJH55MG4ARTKENXN6ENSAB6SO2VXXCXF7CSGS6PQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.529310872 (166h33m28.668295066s) [noise=true , address=145.79.227.16:4914 , piece_id=LOH6G33NOTSPX2EHGYPSEXHCR2VLUWRR2BYRZO5KATQSN3KUTTZA, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.531061212 (166h33m28.670045397s) [noise=true , address=145.79.227.35:4486 , piece_id=D6SHML7OT3YNTWYLFOQWBDLUZCBOE3HIU4E7N2F7SLGOGCUQ5RMQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.533467337 (166h33m28.672451532s) [noise=true , address=145.79.227.11:4731 , piece_id=MX5CYRWIYNI55NDKVMWGA3AFS2LMUPUU34246CNIBA24BWDVARIQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.540788244 (166h33m28.679772438s) [noise=true , address=145.79.227.26:4653 , piece_id=NR3KMDIWN4WZM7ASUFHNBJJZPOI5IZJJLWTEZO6IYMRYMA4IO2TQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.541335998 (166h33m28.680320192s) [noise=true , address=145.79.227.20:4443 , piece_id=MYRORKSVAKD76XH6EJ7NVP5P5YNQ7TLESLF2LAA22B6PH5SHTQCQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.542170426 (166h33m28.68115462s) [noise=true , address=145.79.227.40:4049 , piece_id=NPRJTV5UJRTE52IDTVHMFXJD7CYR2I5C2ZFLUJVAVD6ZUZMVZNAQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.543683784 (166h33m28.682667968s) [noise=true , address=77.90.31.18:4805 , piece_id=6VL6GAR26VPUFXNMERHUYB746PA72NZDA7LTJU3CHSRGV6EYPWDQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.545376767 (166h33m28.684360961s) [noise=true , address=45.91.116.5:4297 , piece_id=WZJLLLQIUTPKW247TATRLH5K425MX2ZD76JNGJQT4KME7AHV2FFA, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.546176590 (166h33m28.685160784s) [noise=true , address=145.79.227.38:4978 , piece_id=DZ3YISF75444FUUJGNDVG3SPZSWOIRJEVZYAKTCG73D7KIYXOXRQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.549820388 (166h33m28.688804572s) [noise=true , address=145.79.227.2:4114 , piece_id=RRA45YDH5ZDU4CH5CTNQMKWPFENHYDEGE7V3NKLJ3LRXGFQORL5Q, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.549827611 (166h33m28.688811795s) [seg_pos={0 0}, upload=146474]: Satisfied cohort requirements after 13 pieces
Feb 24 18:42:09.549977892 (166h33m28.688962086s) [noise=true , address=145.79.227.41:4088 , piece_id=S25UWWCAVJUXYEGMU45PHVTQX4L5BRFAKEIHDQD426UYI5XOEZIA, seg_pos={0 0}, upload=146474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj): protocol: expected piece hash; context canceled; EOF
Feb 24 18:42:09.549983733 (166h33m28.688967917s) [noise=true , address=145.79.227.41:4088 , piece_id=S25UWWCAVJUXYEGMU45PHVTQX4L5BRFAKEIHDQD426UYI5XOEZIA, seg_pos={0 0}, upload=146474]: Long tail cancellation due to optimal threshold
Feb 24 18:42:09.550007498 (166h33m28.688991692s) [noise=true , address=145.79.227.43:4157 , piece_id=WM3UGYNXPAYIWMZT3DXFLV44LWGMAKHN7YUWG4A2HOO26HSPCGKQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF
Feb 24 18:42:09.550013860 (166h33m28.688998044s) [noise=true , address=145.79.227.43:4157 , piece_id=WM3UGYNXPAYIWMZT3DXFLV44LWGMAKHN7YUWG4A2HOO26HSPCGKQ, seg_pos={0 0}, upload=146474]: Long tail cancellation due to optimal threshold
Feb 24 18:42:09.550076105 (166h33m28.689060299s) [noise=true , address=145.79.227.37:4567 , piece_id=KLDMATQWJ5BJBN6EH76Q27YKWK6MFT74AAR543RYW2NX6YBHCO7Q, seg_pos={0 0}, upload=146474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF
Feb 24 18:42:09.550081025 (166h33m28.689065219s) [noise=true , address=145.79.227.37:4567 , piece_id=KLDMATQWJ5BJBN6EH76Q27YKWK6MFT74AAR543RYW2NX6YBHCO7Q, seg_pos={0 0}, upload=146474]: Long tail cancellation due to optimal threshold
Feb 24 18:42:09.550112364 (166h33m28.689096558s) [seg_pos={0 0}, upload=146474]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 24 18:42:09.550210447 (166h33m28.689194631s) [upload=146474]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 24 18:42:09.550214775 (166h33m28.689198959s) [upload=146474]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 24 18:43:51.977860968 (166h35m11.116845182s) [upload=146474]: Done uploading segments...
Feb 24 18:43:51.977945004 (166h35m11.116929219s) [upload=146474]: Upload finished. err: metaclient: context deadline exceeded
Duration: 395.636231ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
42 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..........] duration:
965.122092ms trace:
https://jaeger.dev.storj.io/trace/6973a18e73267103
FAIL (2026-02-24 18:50:32 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 965.144485ms
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
26.89747ms remote:
"tcp:136.0.77.3:443" response headers:
Content-Security-Policy=block-all-mixed-content Content-Type=application/xml X-Amz-Request-Id=18974356DCAE8B58 Date=Tue, 24 Feb 2026 18:50:18 GMT Accept-Ranges=bytes Content-Length=274 Server=Storj Vary=Origin X-Request-Id=fhJ49wLwTSW X-Xss-Protection=1; mode=block target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011 trace:
https://jaeger.dev.storj.io/trace/fa1577c298eef28
FAIL (2026-02-24 18:50:18 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>18974356DCAE8B58</RequestId><HostId></HostId></Error>"
Duration: 26.9191ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
25.45301ms remote:
"tcp:136.0.77.3:443" response headers:
Accept-Ranges=bytes Content-Length=274 Content-Security-Policy=block-all-mixed-content Server=Storj Vary=Origin X-Amz-Request-Id=189743561D3A50D3 X-Xss-Protection=1; mode=block Date=Tue, 24 Feb 2026 18:50:15 GMT Content-Type=application/xml X-Request-Id=hvc8QqnHaAB target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/334cbd9e77d776a1
FAIL (2026-02-24 18:50:15 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>189743561D3A50D3</RequestId><HostId></HostId></Error>"
Duration: 25.475782ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
40 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XX.....] duration:
1.745762253s trace:
https://jaeger.dev.storj.io/trace/c09458d54de96a9
FAIL (2026-02-24 18:50:14 UTC)
operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 1.745784986s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
30 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..........] duration:
639.371778ms trace:
https://jaeger.dev.storj.io/trace/51430d55a27ab7ca
FAIL (2026-02-24 18:50:14 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 639.387688ms
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
15 out of 60 is failed (threshold=6) [.................................XXX.XXXXXX.XXXXXX..........] duration:
60.301925ms remote:
"tcp:136.0.77.4:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/6c3b9ac24855cb84
FAIL (2026-02-24 18:50:13 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 60.311283ms
edge
linksharing
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
29 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........] duration:
150.980996ms trace:
https://jaeger.dev.storj.io/trace/2d0954cf9daf2bbb
FAIL (2026-02-24 18:50:12 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Feb 24 18:42:00.490044390 (166h33m19.629028604s) [upload=146475]: Starting upload Feb 24 18:42:00.490207464 (166h33m19.629191679s) [upload=146475]: Done starting upload Feb 24 18:42:00.490279810 (166h33m19.629264024s) [upload=146475]: Uploading segments... Feb 24 18:42:00.490360300 (166h33m19.629344514s) [upload=146475]: Got next segment. Inline: true Feb 24 18:42:00.490367984 (166h33m19.629352198s) [upload=146475]: Waiting for error group managing segments... Feb 24 18:42:00.490436152 (166h33m19.629420366s) [upload=146475]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Feb 24 18:42:00.490442564 (166h33m19.629426788s) [upload=146475]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Feb 24 18:42:00.490447063 (166h33m19.629431276s) [upload=146475]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Feb 24 18:43:00.472138274 (166h34m19.611122487s) [upload=146475]: Done uploading segments... Feb 24 18:43:00.472188278 (166h34m19.611172491s) [upload=146475]: Upload finished. err: metaclient: context deadline exceeded
Duration: 150.9989ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [.........X.....................X......X.XXX...X.....X.......] duration:
1.259352758s trace:
https://jaeger.dev.storj.io/trace/29c8bea8f1a363cb
FAIL (2026-02-24 18:50:09 UTC)
operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.259374189s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
41 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X........] duration:
247.025594ms trace:
https://jaeger.dev.storj.io/trace/5b29a0cacadf86f0
FAIL (2026-02-24 18:50:04 UTC)
operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Feb 24 18:44:31.418187254 (166h35m50.557171448s) [upload=146507]: Starting upload Feb 24 18:44:31.418228731 (166h35m50.557212916s) [upload=146507]: Done starting upload Feb 24 18:44:31.418274858 (166h35m50.557259042s) [upload=146507]: Uploading segments... Feb 24 18:44:31.418305145 (166h35m50.557289339s) [upload=146507]: Got next segment. Inline: true Feb 24 18:44:31.418308962 (166h35m50.557293156s) [upload=146507]: Waiting for error group managing segments... Feb 24 18:44:31.418363463 (166h35m50.557347657s) [upload=146507]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Feb 24 18:44:31.418369304 (166h35m50.557353488s) [upload=146507]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Feb 24 18:44:31.418372259 (166h35m50.557356443s) [upload=146507]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Feb 24 18:44:50.941040072 (166h36m10.080024286s) [upload=146507]: Done uploading segments... Feb 24 18:44:50.941058296 (166h36m10.08004251s) [upload=146507]: Upload finished. err: <nil>
Duration: 247.040512ms
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
7.02346ms remote:
"tcp:45.38.24.2:443" response headers:
Accept-Ranges=bytes Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1897435350E18C93 X-Request-Id=RRscx8mNzaA Content-Length=284 Content-Security-Policy=block-all-mixed-content Server=Storj X-Xss-Protection=1; mode=block Date=Tue, 24 Feb 2026 18:50:03 GMT target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/3da0e9ccdcc2daa6
FAIL (2026-02-24 18:50:03 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897435350E18C93</RequestId><HostId></HostId></Error>"
Duration: 7.038267ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
25.486201ms remote:
"tcp:136.0.77.3:443" response headers:
Accept-Ranges=bytes Content-Length=274 Content-Security-Policy=block-all-mixed-content Content-Type=application/xml Server=Storj X-Amz-Request-Id=1897435327001A1D X-Request-Id=3yfTCRo1evm Date=Tue, 24 Feb 2026 18:50:02 GMT Vary=Origin X-Xss-Protection=1; mode=block target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/4d91c0260eb2a147
FAIL (2026-02-24 18:50:02 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897435327001A1D</RequestId><HostId></HostId></Error>"
Duration: 25.513753ms
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXX........] duration:
363.125075ms trace:
https://jaeger.dev.storj.io/trace/7b154e70da6a7540
FAIL (2026-02-24 18:49:56 UTC)
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Feb 24 18:41:57.999131463 (166h33m17.138115677s) [upload=146474]: Starting upload
Feb 24 18:41:57.999198959 (166h33m17.138183174s) [upload=146474]: Done starting upload
Feb 24 18:41:57.999328141 (166h33m17.138312335s) [upload=146474]: Uploading segments...
Feb 24 18:41:57.999615107 (166h33m17.138599331s) [upload=146474]: Got next segment. Inline: false
Feb 24 18:41:57.999672624 (166h33m17.138656837s) [upload=146474]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 24 18:41:57.999678475 (166h33m17.138662688s) [upload=146474]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 24 18:42:09.429844419 (166h33m28.568828603s) [seg_pos={0 0}, upload=146474]: Begin upload segment...
Feb 24 18:42:09.429878984 (166h33m28.568863178s) [seg_pos={0 0}, upload=146474]: Done begin upload segment.
Feb 24 18:42:09.429901105 (166h33m28.568885299s) [noise=true , address=145.79.227.11:4731 , piece_id=MX5CYRWIYNI55NDKVMWGA3AFS2LMUPUU34246CNIBA24BWDVARIQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.429935218 (166h33m28.568919403s) [noise=true , address=145.79.227.35:4486 , piece_id=D6SHML7OT3YNTWYLFOQWBDLUZCBOE3HIU4E7N2F7SLGOGCUQ5RMQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.430230660 (166h33m28.569214854s) [noise=true , address=145.79.227.15:4875 , piece_id=FXYLYOG4Y2MQ5NKOYUVPKZU7NOIECZGBAQT5B5WTYGHJ7TQ54C4A, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.430492840 (166h33m28.569477024s) [noise=true , address=145.79.227.43:4157 , piece_id=WM3UGYNXPAYIWMZT3DXFLV44LWGMAKHN7YUWG4A2HOO26HSPCGKQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.430750783 (166h33m28.569734967s) [noise=true , address=145.79.227.16:4914 , piece_id=LOH6G33NOTSPX2EHGYPSEXHCR2VLUWRR2BYRZO5KATQSN3KUTTZA, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.430991071 (166h33m28.569975256s) [noise=true , address=145.79.227.49:4343 , piece_id=HJMNJUJ62JDEKJH55MG4ARTKENXN6ENSAB6SO2VXXCXF7CSGS6PQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.431233163 (166h33m28.570217357s) [noise=true , address=45.91.116.5:4297 , piece_id=WZJLLLQIUTPKW247TATRLH5K425MX2ZD76JNGJQT4KME7AHV2FFA, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.431551629 (166h33m28.570535813s) [noise=true , address=145.79.227.37:4567 , piece_id=KLDMATQWJ5BJBN6EH76Q27YKWK6MFT74AAR543RYW2NX6YBHCO7Q, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.431839126 (166h33m28.57082332s) [noise=true , address=145.79.227.20:4443 , piece_id=MYRORKSVAKD76XH6EJ7NVP5P5YNQ7TLESLF2LAA22B6PH5SHTQCQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.433266051 (166h33m28.572250245s) [noise=true , address=145.79.227.41:4088 , piece_id=S25UWWCAVJUXYEGMU45PHVTQX4L5BRFAKEIHDQD426UYI5XOEZIA, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.434446958 (166h33m28.573431142s) [noise=true , address=145.79.227.39:4018 , piece_id=3QEOPGVK32JLEJV2P4HVZY763WAZCMBKFKRKYKAKW2RUZIAUXODQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.435448058 (166h33m28.574432252s) [noise=true , address=77.90.31.18:4805 , piece_id=6VL6GAR26VPUFXNMERHUYB746PA72NZDA7LTJU3CHSRGV6EYPWDQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.436699445 (166h33m28.575683639s) [noise=true , address=145.79.227.38:4978 , piece_id=DZ3YISF75444FUUJGNDVG3SPZSWOIRJEVZYAKTCG73D7KIYXOXRQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.438093540 (166h33m28.577077734s) [noise=true , address=145.79.227.26:4653 , piece_id=NR3KMDIWN4WZM7ASUFHNBJJZPOI5IZJJLWTEZO6IYMRYMA4IO2TQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.439307297 (166h33m28.578291481s) [noise=true , address=145.79.227.2:4114 , piece_id=RRA45YDH5ZDU4CH5CTNQMKWPFENHYDEGE7V3NKLJ3LRXGFQORL5Q, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.439761746 (166h33m28.57874594s) [noise=true , address=145.79.227.40:4049 , piece_id=NPRJTV5UJRTE52IDTVHMFXJD7CYR2I5C2ZFLUJVAVD6ZUZMVZNAQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.518930921 (166h33m28.657915115s) [upload=146474]: Next returned nil segment
Feb 24 18:42:09.518940428 (166h33m28.657924612s) [upload=146474]: Waiting for error group managing segments...
Feb 24 18:42:09.528008379 (166h33m28.666992563s) [noise=true , address=145.79.227.15:4875 , piece_id=FXYLYOG4Y2MQ5NKOYUVPKZU7NOIECZGBAQT5B5WTYGHJ7TQ54C4A, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.528221628 (166h33m28.667205812s) [noise=true , address=145.79.227.39:4018 , piece_id=3QEOPGVK32JLEJV2P4HVZY763WAZCMBKFKRKYKAKW2RUZIAUXODQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.528377569 (166h33m28.667361753s) [noise=true , address=145.79.227.49:4343 , piece_id=HJMNJUJ62JDEKJH55MG4ARTKENXN6ENSAB6SO2VXXCXF7CSGS6PQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.529310872 (166h33m28.668295066s) [noise=true , address=145.79.227.16:4914 , piece_id=LOH6G33NOTSPX2EHGYPSEXHCR2VLUWRR2BYRZO5KATQSN3KUTTZA, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.531061212 (166h33m28.670045397s) [noise=true , address=145.79.227.35:4486 , piece_id=D6SHML7OT3YNTWYLFOQWBDLUZCBOE3HIU4E7N2F7SLGOGCUQ5RMQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.533467337 (166h33m28.672451532s) [noise=true , address=145.79.227.11:4731 , piece_id=MX5CYRWIYNI55NDKVMWGA3AFS2LMUPUU34246CNIBA24BWDVARIQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.540788244 (166h33m28.679772438s) [noise=true , address=145.79.227.26:4653 , piece_id=NR3KMDIWN4WZM7ASUFHNBJJZPOI5IZJJLWTEZO6IYMRYMA4IO2TQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.541335998 (166h33m28.680320192s) [noise=true , address=145.79.227.20:4443 , piece_id=MYRORKSVAKD76XH6EJ7NVP5P5YNQ7TLESLF2LAA22B6PH5SHTQCQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.542170426 (166h33m28.68115462s) [noise=true , address=145.79.227.40:4049 , piece_id=NPRJTV5UJRTE52IDTVHMFXJD7CYR2I5C2ZFLUJVAVD6ZUZMVZNAQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.543683784 (166h33m28.682667968s) [noise=true , address=77.90.31.18:4805 , piece_id=6VL6GAR26VPUFXNMERHUYB746PA72NZDA7LTJU3CHSRGV6EYPWDQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.545376767 (166h33m28.684360961s) [noise=true , address=45.91.116.5:4297 , piece_id=WZJLLLQIUTPKW247TATRLH5K425MX2ZD76JNGJQT4KME7AHV2FFA, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.546176590 (166h33m28.685160784s) [noise=true , address=145.79.227.38:4978 , piece_id=DZ3YISF75444FUUJGNDVG3SPZSWOIRJEVZYAKTCG73D7KIYXOXRQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.549820388 (166h33m28.688804572s) [noise=true , address=145.79.227.2:4114 , piece_id=RRA45YDH5ZDU4CH5CTNQMKWPFENHYDEGE7V3NKLJ3LRXGFQORL5Q, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.549827611 (166h33m28.688811795s) [seg_pos={0 0}, upload=146474]: Satisfied cohort requirements after 13 pieces
Feb 24 18:42:09.549977892 (166h33m28.688962086s) [noise=true , address=145.79.227.41:4088 , piece_id=S25UWWCAVJUXYEGMU45PHVTQX4L5BRFAKEIHDQD426UYI5XOEZIA, seg_pos={0 0}, upload=146474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj): protocol: expected piece hash; context canceled; EOF
Feb 24 18:42:09.549983733 (166h33m28.688967917s) [noise=true , address=145.79.227.41:4088 , piece_id=S25UWWCAVJUXYEGMU45PHVTQX4L5BRFAKEIHDQD426UYI5XOEZIA, seg_pos={0 0}, upload=146474]: Long tail cancellation due to optimal threshold
Feb 24 18:42:09.550007498 (166h33m28.688991692s) [noise=true , address=145.79.227.43:4157 , piece_id=WM3UGYNXPAYIWMZT3DXFLV44LWGMAKHN7YUWG4A2HOO26HSPCGKQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF
Feb 24 18:42:09.550013860 (166h33m28.688998044s) [noise=true , address=145.79.227.43:4157 , piece_id=WM3UGYNXPAYIWMZT3DXFLV44LWGMAKHN7YUWG4A2HOO26HSPCGKQ, seg_pos={0 0}, upload=146474]: Long tail cancellation due to optimal threshold
Feb 24 18:42:09.550076105 (166h33m28.689060299s) [noise=true , address=145.79.227.37:4567 , piece_id=KLDMATQWJ5BJBN6EH76Q27YKWK6MFT74AAR543RYW2NX6YBHCO7Q, seg_pos={0 0}, upload=146474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF
Feb 24 18:42:09.550081025 (166h33m28.689065219s) [noise=true , address=145.79.227.37:4567 , piece_id=KLDMATQWJ5BJBN6EH76Q27YKWK6MFT74AAR543RYW2NX6YBHCO7Q, seg_pos={0 0}, upload=146474]: Long tail cancellation due to optimal threshold
Feb 24 18:42:09.550112364 (166h33m28.689096558s) [seg_pos={0 0}, upload=146474]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 24 18:42:09.550210447 (166h33m28.689194631s) [upload=146474]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 24 18:42:09.550214775 (166h33m28.689198959s) [upload=146474]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 24 18:43:51.977860968 (166h35m11.116845182s) [upload=146474]: Done uploading segments...
Feb 24 18:43:51.977945004 (166h35m11.116929219s) [upload=146474]: Upload finished. err: metaclient: context deadline exceeded
Duration: 363.148589ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
42 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.........] duration:
1.225188276s trace:
https://jaeger.dev.storj.io/trace/6973a18e73267103
FAIL (2026-02-24 18:49:56 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.2252118s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
29 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.........] duration:
1.123504135s trace:
https://jaeger.dev.storj.io/trace/7ea78b408ceac77f
FAIL (2026-02-24 18:49:49 UTC)
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1s) [noise=true , address=79.127.219.41:4948 , piece_id=AKMCPMBJICKLCUO5EDO2VTOGHO3EOBF2ZGUZOVE2JBAJZ7AL4GKQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.919349510 (166h35m20.058333704s) [noise=true , address=79.127.219.46:4900 , piece_id=E6Z43XUEXNCGI76F2Z7WDCK6E5CZMMSBN4NE3VQKT3HAFS2Y7QSQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.919385177 (166h35m20.058369371s) [noise=true , address=45.91.116.13:4179 , piece_id=OQ624U2HU36XD77IKJBP6Z6B3QHEOWW3MLQLOHDTK34N7C25A3FA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.919848713 (166h35m20.058832897s) [noise=true , address=77.90.31.19:4855 , piece_id=ZF7VMBTCLI2YMUMUL3DNJ2UPVIN5WNWM5CVR56T6F3P6LMAAZZCQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920149715 (166h35m20.059133899s) [noise=true , address=45.91.116.6:4985 , piece_id=GGC7CI62BZZB56QJKHL7MZVPBZCNJKGG3XN5JYUD3WB3C3G67XIA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920391487 (166h35m20.059375681s) [noise=true , address=45.91.116.12:4252 , piece_id=ZZ4SDZDF5B7KGQXX2CR6KVKSCCJGWYI327TDMKVEV6VVJNVHFVGA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920646944 (166h35m20.059631128s) [noise=true , address=77.90.31.21:4911 , piece_id=PAC75B4GQR7ZGOONLLG3QZHM62WBGPQGUDJ3QSOHUHBQSGWAVC5Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920909065 (166h35m20.059893248s) [noise=true , address=145.79.227.35:4491 , piece_id=QRWSMM5V6XF75ORBT5E4EONQHZ3BSRKTRJKF5R2OYM7RHWUYOETQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921156867 (166h35m20.060141051s) [noise=true , address=77.90.31.13:4636 , piece_id=UQ5MUJ3MVVLEGB7GH5BXC6XWZY7U2OKSXX3WFOD6OHZYFPOBNKJQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921400382 (166h35m20.060384576s) [noise=true , address=193.149.164.66:4972 , piece_id=22NWJODMK6GYWQQRUGQDTR7AMOBFXSNK5VE26N75OKEY36YS2CVA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921659165 (166h35m20.060643349s) [noise=true , address=45.91.116.8:4052 , piece_id=OSFGARHZ4RC3GB6TGL57OUS6G5HML7HBLEPVMQQM3HWJO6UBTA7Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921973843 (166h35m20.060958037s) [noise=true , address=77.90.31.16:4742 , piece_id=UQLVEP6CSK7AQNGD63UP2CPV7AIA2NIZ4RZXCPOZXJPOBRM7TH6Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.922281628 (166h35m20.061265822s) [noise=true , address=145.79.227.34:4939 , piece_id=KFYQXKIMP5FUYCXO5ADS322HOPEBT33QZ7ZBIYXLUGEXMVVEXLMA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.922631973 (166h35m20.061616157s) [noise=true , address=145.79.227.37:4582 , piece_id=3KJKXV6QUMY2WXQQ65MF6PR3IKXQ6BGHPBOCCRMBG3BPGCSQIXEQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.924956486 (166h35m20.06394068s) [noise=true , address=145.79.227.14:4846 , piece_id=QXT2YVIRCQT7HKZKTJR4BRQRNTWGAFYVQF4YZEIKCRA5YY77ZLJA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.926527841 (166h35m20.065512036s) [noise=true , address=145.79.227.29:4970 , piece_id=KENFISHR4HYVPMBMD33G2E5G6LARJZN3NQYOZ4KKJ7ODMLNZQT2Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.928105218 (166h35m20.067089402s) [noise=true , address=45.91.116.14:4204 , piece_id=XQQZ4DIUCLHLQILPOYN2RL7ON5VRJCFS622OJC6Y4Y7RSG6B4W7Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.929764789 (166h35m20.068748983s) [noise=true , address=77.90.31.14:4689 , piece_id=FS3QNBO7K3N3ZIZYPV57AAIMJQ33NKCGT4GBCA7THC5QDEO7YEQQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:01.012045757 (166h35m20.151029971s) [upload=146493]: Next returned nil segment
Feb 24 18:44:01.012057469 (166h35m20.151041683s) [upload=146493]: Waiting for error group managing segments...
Feb 24 18:44:01.020560936 (166h35m20.15954515s) [noise=true , address=45.91.116.8:4052 , piece_id=OSFGARHZ4RC3GB6TGL57OUS6G5HML7HBLEPVMQQM3HWJO6UBTA7Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.022333197 (166h35m20.161317401s) [noise=true , address=79.127.219.46:4900 , piece_id=E6Z43XUEXNCGI76F2Z7WDCK6E5CZMMSBN4NE3VQKT3HAFS2Y7QSQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.035953132 (166h35m20.174937356s) [noise=true , address=145.79.227.35:4491 , piece_id=QRWSMM5V6XF75ORBT5E4EONQHZ3BSRKTRJKF5R2OYM7RHWUYOETQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.036725164 (166h35m20.175709388s) [noise=true , address=45.91.116.12:4252 , piece_id=ZZ4SDZDF5B7KGQXX2CR6KVKSCCJGWYI327TDMKVEV6VVJNVHFVGA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.041227315 (166h35m20.180211529s) [noise=true , address=45.91.116.6:4985 , piece_id=GGC7CI62BZZB56QJKHL7MZVPBZCNJKGG3XN5JYUD3WB3C3G67XIA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.044901219 (166h35m20.183885433s) [noise=true , address=45.91.116.14:4204 , piece_id=XQQZ4DIUCLHLQILPOYN2RL7ON5VRJCFS622OJC6Y4Y7RSG6B4W7Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.046613308 (166h35m20.185597521s) [noise=true , address=77.90.31.16:4742 , piece_id=UQLVEP6CSK7AQNGD63UP2CPV7AIA2NIZ4RZXCPOZXJPOBRM7TH6Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.049905900 (166h35m20.188890123s) [noise=true , address=77.90.31.19:4855 , piece_id=ZF7VMBTCLI2YMUMUL3DNJ2UPVIN5WNWM5CVR56T6F3P6LMAAZZCQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.050217330 (166h35m20.189201544s) [noise=true , address=77.90.31.13:4636 , piece_id=UQ5MUJ3MVVLEGB7GH5BXC6XWZY7U2OKSXX3WFOD6OHZYFPOBNKJQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.063062278 (166h35m20.202046492s) [noise=true , address=77.90.31.14:4689 , piece_id=FS3QNBO7K3N3ZIZYPV57AAIMJQ33NKCGT4GBCA7THC5QDEO7YEQQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.063877701 (166h35m20.202861915s) [noise=true , address=45.91.116.13:4179 , piece_id=OQ624U2HU36XD77IKJBP6Z6B3QHEOWW3MLQLOHDTK34N7C25A3FA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.066973164 (166h35m20.205957379s) [noise=true , address=77.90.31.21:4911 , piece_id=PAC75B4GQR7ZGOONLLG3QZHM62WBGPQGUDJ3QSOHUHBQSGWAVC5Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.093746163 (166h35m20.232730377s) [noise=true , address=145.79.227.14:4846 , piece_id=QXT2YVIRCQT7HKZKTJR4BRQRNTWGAFYVQF4YZEIKCRA5YY77ZLJA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.099012601 (166h35m20.237996825s) [noise=true , address=145.79.227.29:4970 , piece_id=KENFISHR4HYVPMBMD33G2E5G6LARJZN3NQYOZ4KKJ7ODMLNZQT2Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.099029282 (166h35m20.238013496s) [seg_pos={0 0}, upload=146493]: Satisfied cohort requirements after 14 pieces
Feb 24 18:44:01.099459185 (166h35m20.2384434s) [noise=true , address=79.127.219.41:4948 , piece_id=AKMCPMBJICKLCUO5EDO2VTOGHO3EOBF2ZGUZOVE2JBAJZ7AL4GKQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gXNjKMdu97i8BC5fYWMiMf3FTepsmHsViuksBWNVLpraTJ1XW): protocol: expected piece hash; context canceled; EOF
Feb 24 18:44:01.099506874 (166h35m20.238491089s) [noise=true , address=79.127.219.41:4948 , piece_id=AKMCPMBJICKLCUO5EDO2VTOGHO3EOBF2ZGUZOVE2JBAJZ7AL4GKQ, seg_pos={0 0}, upload=146493]: Long tail cancellation due to optimal threshold
Feb 24 18:44:01.099675470 (166h35m20.238659684s) [noise=true , address=145.79.227.37:4582 , piece_id=3KJKXV6QUMY2WXQQ65MF6PR3IKXQ6BGHPBOCCRMBG3BPGCSQIXEQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn): protocol: expected piece hash; context canceled; EOF
Feb 24 18:44:01.099683645 (166h35m20.238667859s) [noise=true , address=145.79.227.37:4582 , piece_id=3KJKXV6QUMY2WXQQ65MF6PR3IKXQ6BGHPBOCCRMBG3BPGCSQIXEQ, seg_pos={0 0}, upload=146493]: Long tail cancellation due to optimal threshold
Feb 24 18:44:01.099865534 (166h35m20.238849749s) [noise=true , address=193.149.164.66:4972 , piece_id=22NWJODMK6GYWQQRUGQDTR7AMOBFXSNK5VE26N75OKEY36YS2CVA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128eDneuvfeZbqFp8YQhWHLb7hWrw7T3a2EJ7tzZkkYhaNq7CMT): protocol: read tcp 5.161.89.252:24006->193.149.164.66:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 24 18:44:01.099872908 (166h35m20.238857132s) [noise=true , address=193.149.164.66:4972 , piece_id=22NWJODMK6GYWQQRUGQDTR7AMOBFXSNK5VE26N75OKEY36YS2CVA, seg_pos={0 0}, upload=146493]: Long tail cancellation due to optimal threshold
Feb 24 18:44:01.100082470 (166h35m20.239066684s) [noise=true , address=145.79.227.34:4939 , piece_id=KFYQXKIMP5FUYCXO5ADS322HOPEBT33QZ7ZBIYXLUGEXMVVEXLMA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.100105593 (166h35m20.239089808s) [seg_pos={0 0}, upload=146493]: Satisfied cohort requirements after 15 pieces
Feb 24 18:44:01.100170554 (166h35m20.239154778s) [seg_pos={0 0}, upload=146493]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Feb 24 18:44:01.100364717 (166h35m20.239348941s) [upload=146493]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 24 18:44:01.100379766 (166h35m20.23936399s) [upload=146493]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 24 18:44:14.739172222 (166h35m33.878156437s) [upload=146493]: Done uploading segments...
Feb 24 18:44:14.739199784 (166h35m33.878184008s) [upload=146493]: Upload finished. err: <nil>
Duration: 1.123525114s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
43 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.......] duration:
149.448593ms trace:
https://jaeger.dev.storj.io/trace/5383e9cc16a1cf07
FAIL (2026-02-24 18:49:49 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 149.476355ms
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
41 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.......] duration:
206.592147ms trace:
https://jaeger.dev.storj.io/trace/5b29a0cacadf86f0
FAIL (2026-02-24 18:49:45 UTC)
operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Feb 24 18:44:31.418187254 (166h35m50.557171448s) [upload=146507]: Starting upload Feb 24 18:44:31.418228731 (166h35m50.557212916s) [upload=146507]: Done starting upload Feb 24 18:44:31.418274858 (166h35m50.557259042s) [upload=146507]: Uploading segments... Feb 24 18:44:31.418305145 (166h35m50.557289339s) [upload=146507]: Got next segment. Inline: true Feb 24 18:44:31.418308962 (166h35m50.557293156s) [upload=146507]: Waiting for error group managing segments... Feb 24 18:44:31.418363463 (166h35m50.557347657s) [upload=146507]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Feb 24 18:44:31.418369304 (166h35m50.557353488s) [upload=146507]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Feb 24 18:44:31.418372259 (166h35m50.557356443s) [upload=146507]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Feb 24 18:44:50.941040072 (166h36m10.080024286s) [upload=146507]: Done uploading segments... Feb 24 18:44:50.941058296 (166h36m10.08004251s) [upload=146507]: Upload finished. err: <nil>
Duration: 206.612755ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
29 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX..........] duration:
193.531246ms trace:
https://jaeger.dev.storj.io/trace/2d0954cf9daf2bbb
FAIL (2026-02-24 18:49:42 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Feb 24 18:42:00.490044390 (166h33m19.629028604s) [upload=146475]: Starting upload Feb 24 18:42:00.490207464 (166h33m19.629191679s) [upload=146475]: Done starting upload Feb 24 18:42:00.490279810 (166h33m19.629264024s) [upload=146475]: Uploading segments... Feb 24 18:42:00.490360300 (166h33m19.629344514s) [upload=146475]: Got next segment. Inline: true Feb 24 18:42:00.490367984 (166h33m19.629352198s) [upload=146475]: Waiting for error group managing segments... Feb 24 18:42:00.490436152 (166h33m19.629420366s) [upload=146475]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Feb 24 18:42:00.490442564 (166h33m19.629426788s) [upload=146475]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Feb 24 18:42:00.490447063 (166h33m19.629431276s) [upload=146475]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Feb 24 18:43:00.472138274 (166h34m19.611122487s) [upload=146475]: Done uploading segments... Feb 24 18:43:00.472188278 (166h34m19.611172491s) [upload=146475]: Upload finished. err: metaclient: context deadline exceeded
Duration: 193.549531ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
44 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX......] duration:
392.503889ms trace:
https://jaeger.dev.storj.io/trace/4c427ad68ce4a878
FAIL (2026-02-24 18:49:42 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 392.526211ms
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
29.80535ms remote:
"tcp:136.0.77.3:443" response headers:
Server=Storj X-Amz-Request-Id=1897434D1543FCFE X-Request-Id=9wA9AkUMhZf X-Xss-Protection=1; mode=block Vary=Origin Date=Tue, 24 Feb 2026 18:49:36 GMT Accept-Ranges=bytes Content-Length=274 Content-Security-Policy=block-all-mixed-content Content-Type=application/xml target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011 trace:
https://jaeger.dev.storj.io/trace/3f0ac0a5615822f3
FAIL (2026-02-24 18:49:36 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897434D1543FCFE</RequestId><HostId></HostId></Error>"
Duration: 29.827402ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
11.579893ms remote:
"tcp:45.38.24.2:443" response headers:
Vary=Origin X-Amz-Request-Id=1897434C0EBC553F X-Request-Id=CvERBeMGCvX Content-Length=284 Server=Storj X-Xss-Protection=1; mode=block Date=Tue, 24 Feb 2026 18:49:32 GMT Accept-Ranges=bytes Content-Security-Policy=block-all-mixed-content Content-Type=application/xml target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/597a7dcb1ddb1e5e
FAIL (2026-02-24 18:49:32 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>1897434C0EBC553F</RequestId><HostId></HostId></Error>"
Duration: 11.597174ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [..........X.....................X......X.XXX...X.....X......] duration:
1.087546469s trace:
https://jaeger.dev.storj.io/trace/29c8bea8f1a363cb
FAIL (2026-02-24 18:49:25 UTC)
operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.087563652s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
30 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.........] duration:
640.883833ms trace:
https://jaeger.dev.storj.io/trace/51430d55a27ab7ca
FAIL (2026-02-24 18:49:25 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 640.904111ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
42 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX........] duration:
678.545221ms trace:
https://jaeger.dev.storj.io/trace/6973a18e73267103
FAIL (2026-02-24 18:49:24 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 678.571822ms
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
15 out of 60 is failed (threshold=6) [..................................XXX.XXXXXX.XXXXXX.........] duration:
42.722142ms remote:
"tcp:136.0.77.4:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/6c3b9ac24855cb84
FAIL (2026-02-24 18:49:13 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 42.740316ms
edge
linksharing
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
29 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX........] duration:
1.121969895s trace:
https://jaeger.dev.storj.io/trace/7ea78b408ceac77f
FAIL (2026-02-24 18:49:09 UTC)
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1s) [noise=true , address=79.127.219.41:4948 , piece_id=AKMCPMBJICKLCUO5EDO2VTOGHO3EOBF2ZGUZOVE2JBAJZ7AL4GKQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.919349510 (166h35m20.058333704s) [noise=true , address=79.127.219.46:4900 , piece_id=E6Z43XUEXNCGI76F2Z7WDCK6E5CZMMSBN4NE3VQKT3HAFS2Y7QSQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.919385177 (166h35m20.058369371s) [noise=true , address=45.91.116.13:4179 , piece_id=OQ624U2HU36XD77IKJBP6Z6B3QHEOWW3MLQLOHDTK34N7C25A3FA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.919848713 (166h35m20.058832897s) [noise=true , address=77.90.31.19:4855 , piece_id=ZF7VMBTCLI2YMUMUL3DNJ2UPVIN5WNWM5CVR56T6F3P6LMAAZZCQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920149715 (166h35m20.059133899s) [noise=true , address=45.91.116.6:4985 , piece_id=GGC7CI62BZZB56QJKHL7MZVPBZCNJKGG3XN5JYUD3WB3C3G67XIA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920391487 (166h35m20.059375681s) [noise=true , address=45.91.116.12:4252 , piece_id=ZZ4SDZDF5B7KGQXX2CR6KVKSCCJGWYI327TDMKVEV6VVJNVHFVGA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920646944 (166h35m20.059631128s) [noise=true , address=77.90.31.21:4911 , piece_id=PAC75B4GQR7ZGOONLLG3QZHM62WBGPQGUDJ3QSOHUHBQSGWAVC5Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920909065 (166h35m20.059893248s) [noise=true , address=145.79.227.35:4491 , piece_id=QRWSMM5V6XF75ORBT5E4EONQHZ3BSRKTRJKF5R2OYM7RHWUYOETQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921156867 (166h35m20.060141051s) [noise=true , address=77.90.31.13:4636 , piece_id=UQ5MUJ3MVVLEGB7GH5BXC6XWZY7U2OKSXX3WFOD6OHZYFPOBNKJQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921400382 (166h35m20.060384576s) [noise=true , address=193.149.164.66:4972 , piece_id=22NWJODMK6GYWQQRUGQDTR7AMOBFXSNK5VE26N75OKEY36YS2CVA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921659165 (166h35m20.060643349s) [noise=true , address=45.91.116.8:4052 , piece_id=OSFGARHZ4RC3GB6TGL57OUS6G5HML7HBLEPVMQQM3HWJO6UBTA7Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921973843 (166h35m20.060958037s) [noise=true , address=77.90.31.16:4742 , piece_id=UQLVEP6CSK7AQNGD63UP2CPV7AIA2NIZ4RZXCPOZXJPOBRM7TH6Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.922281628 (166h35m20.061265822s) [noise=true , address=145.79.227.34:4939 , piece_id=KFYQXKIMP5FUYCXO5ADS322HOPEBT33QZ7ZBIYXLUGEXMVVEXLMA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.922631973 (166h35m20.061616157s) [noise=true , address=145.79.227.37:4582 , piece_id=3KJKXV6QUMY2WXQQ65MF6PR3IKXQ6BGHPBOCCRMBG3BPGCSQIXEQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.924956486 (166h35m20.06394068s) [noise=true , address=145.79.227.14:4846 , piece_id=QXT2YVIRCQT7HKZKTJR4BRQRNTWGAFYVQF4YZEIKCRA5YY77ZLJA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.926527841 (166h35m20.065512036s) [noise=true , address=145.79.227.29:4970 , piece_id=KENFISHR4HYVPMBMD33G2E5G6LARJZN3NQYOZ4KKJ7ODMLNZQT2Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.928105218 (166h35m20.067089402s) [noise=true , address=45.91.116.14:4204 , piece_id=XQQZ4DIUCLHLQILPOYN2RL7ON5VRJCFS622OJC6Y4Y7RSG6B4W7Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.929764789 (166h35m20.068748983s) [noise=true , address=77.90.31.14:4689 , piece_id=FS3QNBO7K3N3ZIZYPV57AAIMJQ33NKCGT4GBCA7THC5QDEO7YEQQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:01.012045757 (166h35m20.151029971s) [upload=146493]: Next returned nil segment
Feb 24 18:44:01.012057469 (166h35m20.151041683s) [upload=146493]: Waiting for error group managing segments...
Feb 24 18:44:01.020560936 (166h35m20.15954515s) [noise=true , address=45.91.116.8:4052 , piece_id=OSFGARHZ4RC3GB6TGL57OUS6G5HML7HBLEPVMQQM3HWJO6UBTA7Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.022333197 (166h35m20.161317401s) [noise=true , address=79.127.219.46:4900 , piece_id=E6Z43XUEXNCGI76F2Z7WDCK6E5CZMMSBN4NE3VQKT3HAFS2Y7QSQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.035953132 (166h35m20.174937356s) [noise=true , address=145.79.227.35:4491 , piece_id=QRWSMM5V6XF75ORBT5E4EONQHZ3BSRKTRJKF5R2OYM7RHWUYOETQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.036725164 (166h35m20.175709388s) [noise=true , address=45.91.116.12:4252 , piece_id=ZZ4SDZDF5B7KGQXX2CR6KVKSCCJGWYI327TDMKVEV6VVJNVHFVGA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.041227315 (166h35m20.180211529s) [noise=true , address=45.91.116.6:4985 , piece_id=GGC7CI62BZZB56QJKHL7MZVPBZCNJKGG3XN5JYUD3WB3C3G67XIA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.044901219 (166h35m20.183885433s) [noise=true , address=45.91.116.14:4204 , piece_id=XQQZ4DIUCLHLQILPOYN2RL7ON5VRJCFS622OJC6Y4Y7RSG6B4W7Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.046613308 (166h35m20.185597521s) [noise=true , address=77.90.31.16:4742 , piece_id=UQLVEP6CSK7AQNGD63UP2CPV7AIA2NIZ4RZXCPOZXJPOBRM7TH6Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.049905900 (166h35m20.188890123s) [noise=true , address=77.90.31.19:4855 , piece_id=ZF7VMBTCLI2YMUMUL3DNJ2UPVIN5WNWM5CVR56T6F3P6LMAAZZCQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.050217330 (166h35m20.189201544s) [noise=true , address=77.90.31.13:4636 , piece_id=UQ5MUJ3MVVLEGB7GH5BXC6XWZY7U2OKSXX3WFOD6OHZYFPOBNKJQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.063062278 (166h35m20.202046492s) [noise=true , address=77.90.31.14:4689 , piece_id=FS3QNBO7K3N3ZIZYPV57AAIMJQ33NKCGT4GBCA7THC5QDEO7YEQQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.063877701 (166h35m20.202861915s) [noise=true , address=45.91.116.13:4179 , piece_id=OQ624U2HU36XD77IKJBP6Z6B3QHEOWW3MLQLOHDTK34N7C25A3FA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.066973164 (166h35m20.205957379s) [noise=true , address=77.90.31.21:4911 , piece_id=PAC75B4GQR7ZGOONLLG3QZHM62WBGPQGUDJ3QSOHUHBQSGWAVC5Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.093746163 (166h35m20.232730377s) [noise=true , address=145.79.227.14:4846 , piece_id=QXT2YVIRCQT7HKZKTJR4BRQRNTWGAFYVQF4YZEIKCRA5YY77ZLJA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.099012601 (166h35m20.237996825s) [noise=true , address=145.79.227.29:4970 , piece_id=KENFISHR4HYVPMBMD33G2E5G6LARJZN3NQYOZ4KKJ7ODMLNZQT2Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.099029282 (166h35m20.238013496s) [seg_pos={0 0}, upload=146493]: Satisfied cohort requirements after 14 pieces
Feb 24 18:44:01.099459185 (166h35m20.2384434s) [noise=true , address=79.127.219.41:4948 , piece_id=AKMCPMBJICKLCUO5EDO2VTOGHO3EOBF2ZGUZOVE2JBAJZ7AL4GKQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gXNjKMdu97i8BC5fYWMiMf3FTepsmHsViuksBWNVLpraTJ1XW): protocol: expected piece hash; context canceled; EOF
Feb 24 18:44:01.099506874 (166h35m20.238491089s) [noise=true , address=79.127.219.41:4948 , piece_id=AKMCPMBJICKLCUO5EDO2VTOGHO3EOBF2ZGUZOVE2JBAJZ7AL4GKQ, seg_pos={0 0}, upload=146493]: Long tail cancellation due to optimal threshold
Feb 24 18:44:01.099675470 (166h35m20.238659684s) [noise=true , address=145.79.227.37:4582 , piece_id=3KJKXV6QUMY2WXQQ65MF6PR3IKXQ6BGHPBOCCRMBG3BPGCSQIXEQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn): protocol: expected piece hash; context canceled; EOF
Feb 24 18:44:01.099683645 (166h35m20.238667859s) [noise=true , address=145.79.227.37:4582 , piece_id=3KJKXV6QUMY2WXQQ65MF6PR3IKXQ6BGHPBOCCRMBG3BPGCSQIXEQ, seg_pos={0 0}, upload=146493]: Long tail cancellation due to optimal threshold
Feb 24 18:44:01.099865534 (166h35m20.238849749s) [noise=true , address=193.149.164.66:4972 , piece_id=22NWJODMK6GYWQQRUGQDTR7AMOBFXSNK5VE26N75OKEY36YS2CVA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128eDneuvfeZbqFp8YQhWHLb7hWrw7T3a2EJ7tzZkkYhaNq7CMT): protocol: read tcp 5.161.89.252:24006->193.149.164.66:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 24 18:44:01.099872908 (166h35m20.238857132s) [noise=true , address=193.149.164.66:4972 , piece_id=22NWJODMK6GYWQQRUGQDTR7AMOBFXSNK5VE26N75OKEY36YS2CVA, seg_pos={0 0}, upload=146493]: Long tail cancellation due to optimal threshold
Feb 24 18:44:01.100082470 (166h35m20.239066684s) [noise=true , address=145.79.227.34:4939 , piece_id=KFYQXKIMP5FUYCXO5ADS322HOPEBT33QZ7ZBIYXLUGEXMVVEXLMA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.100105593 (166h35m20.239089808s) [seg_pos={0 0}, upload=146493]: Satisfied cohort requirements after 15 pieces
Feb 24 18:44:01.100170554 (166h35m20.239154778s) [seg_pos={0 0}, upload=146493]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Feb 24 18:44:01.100364717 (166h35m20.239348941s) [upload=146493]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 24 18:44:01.100379766 (166h35m20.23936399s) [upload=146493]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 24 18:44:14.739172222 (166h35m33.878156437s) [upload=146493]: Done uploading segments...
Feb 24 18:44:14.739199784 (166h35m33.878184008s) [upload=146493]: Upload finished. err: <nil>
Duration: 1.121992888s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
40 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XX....] duration:
1.992503379s trace:
https://jaeger.dev.storj.io/trace/c09458d54de96a9
FAIL (2026-02-24 18:49:04 UTC)
operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 1.992527663s
core
uplink
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
23.491037ms remote:
"tcp:136.0.77.3:443" response headers:
X-Amz-Request-Id=18974345A29139CE X-Request-Id=VC7ibK7ZJXw Content-Security-Policy=block-all-mixed-content Server=Storj X-Xss-Protection=1; mode=block Date=Tue, 24 Feb 2026 18:49:04 GMT Accept-Ranges=bytes Content-Length=274 Content-Type=application/xml Vary=Origin target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/125e9f78b96d8b12
FAIL (2026-02-24 18:49:04 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>18974345A29139CE</RequestId><HostId></HostId></Error>"
Duration: 23.511204ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
41 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X......] duration:
180.826865ms trace:
https://jaeger.dev.storj.io/trace/5b29a0cacadf86f0
FAIL (2026-02-24 18:48:59 UTC)
operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Feb 24 18:44:31.418187254 (166h35m50.557171448s) [upload=146507]: Starting upload Feb 24 18:44:31.418228731 (166h35m50.557212916s) [upload=146507]: Done starting upload Feb 24 18:44:31.418274858 (166h35m50.557259042s) [upload=146507]: Uploading segments... Feb 24 18:44:31.418305145 (166h35m50.557289339s) [upload=146507]: Got next segment. Inline: true Feb 24 18:44:31.418308962 (166h35m50.557293156s) [upload=146507]: Waiting for error group managing segments... Feb 24 18:44:31.418363463 (166h35m50.557347657s) [upload=146507]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Feb 24 18:44:31.418369304 (166h35m50.557353488s) [upload=146507]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Feb 24 18:44:31.418372259 (166h35m50.557356443s) [upload=146507]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Feb 24 18:44:50.941040072 (166h36m10.080024286s) [upload=146507]: Done uploading segments... Feb 24 18:44:50.941058296 (166h36m10.08004251s) [upload=146507]: Upload finished. err: <nil>
Duration: 180.850911ms
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
24.037126ms remote:
"tcp:136.0.77.3:443" response headers:
Accept-Ranges=bytes Server=Storj Vary=Origin X-Request-Id=giHmheAQeR7 Content-Length=274 Content-Security-Policy=block-all-mixed-content Content-Type=application/xml X-Amz-Request-Id=189743437132E5D5 X-Xss-Protection=1; mode=block Date=Tue, 24 Feb 2026 18:48:55 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/563a44e5b66f99ff
FAIL (2026-02-24 18:48:55 UTC)
orbiter warning: unexpected status code "403 Forbidden" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Request has expired</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>189743437132E5D5</RequestId><HostId></HostId></Error>"
Duration: 24.059187ms
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
43 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X......] duration:
161.055095ms trace:
https://jaeger.dev.storj.io/trace/5383e9cc16a1cf07
FAIL (2026-02-24 18:48:54 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 161.076776ms
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
29 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......] duration:
821.511154ms trace:
https://jaeger.dev.storj.io/trace/7ea78b408ceac77f
FAIL (2026-02-24 18:48:49 UTC)
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1s) [noise=true , address=79.127.219.41:4948 , piece_id=AKMCPMBJICKLCUO5EDO2VTOGHO3EOBF2ZGUZOVE2JBAJZ7AL4GKQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.919349510 (166h35m20.058333704s) [noise=true , address=79.127.219.46:4900 , piece_id=E6Z43XUEXNCGI76F2Z7WDCK6E5CZMMSBN4NE3VQKT3HAFS2Y7QSQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.919385177 (166h35m20.058369371s) [noise=true , address=45.91.116.13:4179 , piece_id=OQ624U2HU36XD77IKJBP6Z6B3QHEOWW3MLQLOHDTK34N7C25A3FA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.919848713 (166h35m20.058832897s) [noise=true , address=77.90.31.19:4855 , piece_id=ZF7VMBTCLI2YMUMUL3DNJ2UPVIN5WNWM5CVR56T6F3P6LMAAZZCQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920149715 (166h35m20.059133899s) [noise=true , address=45.91.116.6:4985 , piece_id=GGC7CI62BZZB56QJKHL7MZVPBZCNJKGG3XN5JYUD3WB3C3G67XIA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920391487 (166h35m20.059375681s) [noise=true , address=45.91.116.12:4252 , piece_id=ZZ4SDZDF5B7KGQXX2CR6KVKSCCJGWYI327TDMKVEV6VVJNVHFVGA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920646944 (166h35m20.059631128s) [noise=true , address=77.90.31.21:4911 , piece_id=PAC75B4GQR7ZGOONLLG3QZHM62WBGPQGUDJ3QSOHUHBQSGWAVC5Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.920909065 (166h35m20.059893248s) [noise=true , address=145.79.227.35:4491 , piece_id=QRWSMM5V6XF75ORBT5E4EONQHZ3BSRKTRJKF5R2OYM7RHWUYOETQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921156867 (166h35m20.060141051s) [noise=true , address=77.90.31.13:4636 , piece_id=UQ5MUJ3MVVLEGB7GH5BXC6XWZY7U2OKSXX3WFOD6OHZYFPOBNKJQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921400382 (166h35m20.060384576s) [noise=true , address=193.149.164.66:4972 , piece_id=22NWJODMK6GYWQQRUGQDTR7AMOBFXSNK5VE26N75OKEY36YS2CVA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921659165 (166h35m20.060643349s) [noise=true , address=45.91.116.8:4052 , piece_id=OSFGARHZ4RC3GB6TGL57OUS6G5HML7HBLEPVMQQM3HWJO6UBTA7Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.921973843 (166h35m20.060958037s) [noise=true , address=77.90.31.16:4742 , piece_id=UQLVEP6CSK7AQNGD63UP2CPV7AIA2NIZ4RZXCPOZXJPOBRM7TH6Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.922281628 (166h35m20.061265822s) [noise=true , address=145.79.227.34:4939 , piece_id=KFYQXKIMP5FUYCXO5ADS322HOPEBT33QZ7ZBIYXLUGEXMVVEXLMA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.922631973 (166h35m20.061616157s) [noise=true , address=145.79.227.37:4582 , piece_id=3KJKXV6QUMY2WXQQ65MF6PR3IKXQ6BGHPBOCCRMBG3BPGCSQIXEQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.924956486 (166h35m20.06394068s) [noise=true , address=145.79.227.14:4846 , piece_id=QXT2YVIRCQT7HKZKTJR4BRQRNTWGAFYVQF4YZEIKCRA5YY77ZLJA, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.926527841 (166h35m20.065512036s) [noise=true , address=145.79.227.29:4970 , piece_id=KENFISHR4HYVPMBMD33G2E5G6LARJZN3NQYOZ4KKJ7ODMLNZQT2Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.928105218 (166h35m20.067089402s) [noise=true , address=45.91.116.14:4204 , piece_id=XQQZ4DIUCLHLQILPOYN2RL7ON5VRJCFS622OJC6Y4Y7RSG6B4W7Q, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:00.929764789 (166h35m20.068748983s) [noise=true , address=77.90.31.14:4689 , piece_id=FS3QNBO7K3N3ZIZYPV57AAIMJQ33NKCGT4GBCA7THC5QDEO7YEQQ, seg_pos={0 0}, upload=146493]: Uploading piece...
Feb 24 18:44:01.012045757 (166h35m20.151029971s) [upload=146493]: Next returned nil segment
Feb 24 18:44:01.012057469 (166h35m20.151041683s) [upload=146493]: Waiting for error group managing segments...
Feb 24 18:44:01.020560936 (166h35m20.15954515s) [noise=true , address=45.91.116.8:4052 , piece_id=OSFGARHZ4RC3GB6TGL57OUS6G5HML7HBLEPVMQQM3HWJO6UBTA7Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.022333197 (166h35m20.161317401s) [noise=true , address=79.127.219.46:4900 , piece_id=E6Z43XUEXNCGI76F2Z7WDCK6E5CZMMSBN4NE3VQKT3HAFS2Y7QSQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.035953132 (166h35m20.174937356s) [noise=true , address=145.79.227.35:4491 , piece_id=QRWSMM5V6XF75ORBT5E4EONQHZ3BSRKTRJKF5R2OYM7RHWUYOETQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.036725164 (166h35m20.175709388s) [noise=true , address=45.91.116.12:4252 , piece_id=ZZ4SDZDF5B7KGQXX2CR6KVKSCCJGWYI327TDMKVEV6VVJNVHFVGA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.041227315 (166h35m20.180211529s) [noise=true , address=45.91.116.6:4985 , piece_id=GGC7CI62BZZB56QJKHL7MZVPBZCNJKGG3XN5JYUD3WB3C3G67XIA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.044901219 (166h35m20.183885433s) [noise=true , address=45.91.116.14:4204 , piece_id=XQQZ4DIUCLHLQILPOYN2RL7ON5VRJCFS622OJC6Y4Y7RSG6B4W7Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.046613308 (166h35m20.185597521s) [noise=true , address=77.90.31.16:4742 , piece_id=UQLVEP6CSK7AQNGD63UP2CPV7AIA2NIZ4RZXCPOZXJPOBRM7TH6Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.049905900 (166h35m20.188890123s) [noise=true , address=77.90.31.19:4855 , piece_id=ZF7VMBTCLI2YMUMUL3DNJ2UPVIN5WNWM5CVR56T6F3P6LMAAZZCQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.050217330 (166h35m20.189201544s) [noise=true , address=77.90.31.13:4636 , piece_id=UQ5MUJ3MVVLEGB7GH5BXC6XWZY7U2OKSXX3WFOD6OHZYFPOBNKJQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.063062278 (166h35m20.202046492s) [noise=true , address=77.90.31.14:4689 , piece_id=FS3QNBO7K3N3ZIZYPV57AAIMJQ33NKCGT4GBCA7THC5QDEO7YEQQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.063877701 (166h35m20.202861915s) [noise=true , address=45.91.116.13:4179 , piece_id=OQ624U2HU36XD77IKJBP6Z6B3QHEOWW3MLQLOHDTK34N7C25A3FA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.066973164 (166h35m20.205957379s) [noise=true , address=77.90.31.21:4911 , piece_id=PAC75B4GQR7ZGOONLLG3QZHM62WBGPQGUDJ3QSOHUHBQSGWAVC5Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.093746163 (166h35m20.232730377s) [noise=true , address=145.79.227.14:4846 , piece_id=QXT2YVIRCQT7HKZKTJR4BRQRNTWGAFYVQF4YZEIKCRA5YY77ZLJA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.099012601 (166h35m20.237996825s) [noise=true , address=145.79.227.29:4970 , piece_id=KENFISHR4HYVPMBMD33G2E5G6LARJZN3NQYOZ4KKJ7ODMLNZQT2Q, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.099029282 (166h35m20.238013496s) [seg_pos={0 0}, upload=146493]: Satisfied cohort requirements after 14 pieces
Feb 24 18:44:01.099459185 (166h35m20.2384434s) [noise=true , address=79.127.219.41:4948 , piece_id=AKMCPMBJICKLCUO5EDO2VTOGHO3EOBF2ZGUZOVE2JBAJZ7AL4GKQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gXNjKMdu97i8BC5fYWMiMf3FTepsmHsViuksBWNVLpraTJ1XW): protocol: expected piece hash; context canceled; EOF
Feb 24 18:44:01.099506874 (166h35m20.238491089s) [noise=true , address=79.127.219.41:4948 , piece_id=AKMCPMBJICKLCUO5EDO2VTOGHO3EOBF2ZGUZOVE2JBAJZ7AL4GKQ, seg_pos={0 0}, upload=146493]: Long tail cancellation due to optimal threshold
Feb 24 18:44:01.099675470 (166h35m20.238659684s) [noise=true , address=145.79.227.37:4582 , piece_id=3KJKXV6QUMY2WXQQ65MF6PR3IKXQ6BGHPBOCCRMBG3BPGCSQIXEQ, seg_pos={0 0}, upload=146493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn): protocol: expected piece hash; context canceled; EOF
Feb 24 18:44:01.099683645 (166h35m20.238667859s) [noise=true , address=145.79.227.37:4582 , piece_id=3KJKXV6QUMY2WXQQ65MF6PR3IKXQ6BGHPBOCCRMBG3BPGCSQIXEQ, seg_pos={0 0}, upload=146493]: Long tail cancellation due to optimal threshold
Feb 24 18:44:01.099865534 (166h35m20.238849749s) [noise=true , address=193.149.164.66:4972 , piece_id=22NWJODMK6GYWQQRUGQDTR7AMOBFXSNK5VE26N75OKEY36YS2CVA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128eDneuvfeZbqFp8YQhWHLb7hWrw7T3a2EJ7tzZkkYhaNq7CMT): protocol: read tcp 5.161.89.252:24006->193.149.164.66:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 24 18:44:01.099872908 (166h35m20.238857132s) [noise=true , address=193.149.164.66:4972 , piece_id=22NWJODMK6GYWQQRUGQDTR7AMOBFXSNK5VE26N75OKEY36YS2CVA, seg_pos={0 0}, upload=146493]: Long tail cancellation due to optimal threshold
Feb 24 18:44:01.100082470 (166h35m20.239066684s) [noise=true , address=145.79.227.34:4939 , piece_id=KFYQXKIMP5FUYCXO5ADS322HOPEBT33QZ7ZBIYXLUGEXMVVEXLMA, seg_pos={0 0}, upload=146493]: Done uploading piece. err: <nil>
Feb 24 18:44:01.100105593 (166h35m20.239089808s) [seg_pos={0 0}, upload=146493]: Satisfied cohort requirements after 15 pieces
Feb 24 18:44:01.100170554 (166h35m20.239154778s) [seg_pos={0 0}, upload=146493]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Feb 24 18:44:01.100364717 (166h35m20.239348941s) [upload=146493]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 24 18:44:01.100379766 (166h35m20.23936399s) [upload=146493]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 24 18:44:14.739172222 (166h35m33.878156437s) [upload=146493]: Done uploading segments...
Feb 24 18:44:14.739199784 (166h35m33.878184008s) [upload=146493]: Upload finished. err: <nil>
Duration: 821.534236ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXX.......] duration:
529.500704ms trace:
https://jaeger.dev.storj.io/trace/7b154e70da6a7540
FAIL (2026-02-24 18:48:47 UTC)
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Feb 24 18:41:57.999131463 (166h33m17.138115677s) [upload=146474]: Starting upload
Feb 24 18:41:57.999198959 (166h33m17.138183174s) [upload=146474]: Done starting upload
Feb 24 18:41:57.999328141 (166h33m17.138312335s) [upload=146474]: Uploading segments...
Feb 24 18:41:57.999615107 (166h33m17.138599331s) [upload=146474]: Got next segment. Inline: false
Feb 24 18:41:57.999672624 (166h33m17.138656837s) [upload=146474]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 24 18:41:57.999678475 (166h33m17.138662688s) [upload=146474]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 24 18:42:09.429844419 (166h33m28.568828603s) [seg_pos={0 0}, upload=146474]: Begin upload segment...
Feb 24 18:42:09.429878984 (166h33m28.568863178s) [seg_pos={0 0}, upload=146474]: Done begin upload segment.
Feb 24 18:42:09.429901105 (166h33m28.568885299s) [noise=true , address=145.79.227.11:4731 , piece_id=MX5CYRWIYNI55NDKVMWGA3AFS2LMUPUU34246CNIBA24BWDVARIQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.429935218 (166h33m28.568919403s) [noise=true , address=145.79.227.35:4486 , piece_id=D6SHML7OT3YNTWYLFOQWBDLUZCBOE3HIU4E7N2F7SLGOGCUQ5RMQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.430230660 (166h33m28.569214854s) [noise=true , address=145.79.227.15:4875 , piece_id=FXYLYOG4Y2MQ5NKOYUVPKZU7NOIECZGBAQT5B5WTYGHJ7TQ54C4A, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.430492840 (166h33m28.569477024s) [noise=true , address=145.79.227.43:4157 , piece_id=WM3UGYNXPAYIWMZT3DXFLV44LWGMAKHN7YUWG4A2HOO26HSPCGKQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.430750783 (166h33m28.569734967s) [noise=true , address=145.79.227.16:4914 , piece_id=LOH6G33NOTSPX2EHGYPSEXHCR2VLUWRR2BYRZO5KATQSN3KUTTZA, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.430991071 (166h33m28.569975256s) [noise=true , address=145.79.227.49:4343 , piece_id=HJMNJUJ62JDEKJH55MG4ARTKENXN6ENSAB6SO2VXXCXF7CSGS6PQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.431233163 (166h33m28.570217357s) [noise=true , address=45.91.116.5:4297 , piece_id=WZJLLLQIUTPKW247TATRLH5K425MX2ZD76JNGJQT4KME7AHV2FFA, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.431551629 (166h33m28.570535813s) [noise=true , address=145.79.227.37:4567 , piece_id=KLDMATQWJ5BJBN6EH76Q27YKWK6MFT74AAR543RYW2NX6YBHCO7Q, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.431839126 (166h33m28.57082332s) [noise=true , address=145.79.227.20:4443 , piece_id=MYRORKSVAKD76XH6EJ7NVP5P5YNQ7TLESLF2LAA22B6PH5SHTQCQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.433266051 (166h33m28.572250245s) [noise=true , address=145.79.227.41:4088 , piece_id=S25UWWCAVJUXYEGMU45PHVTQX4L5BRFAKEIHDQD426UYI5XOEZIA, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.434446958 (166h33m28.573431142s) [noise=true , address=145.79.227.39:4018 , piece_id=3QEOPGVK32JLEJV2P4HVZY763WAZCMBKFKRKYKAKW2RUZIAUXODQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.435448058 (166h33m28.574432252s) [noise=true , address=77.90.31.18:4805 , piece_id=6VL6GAR26VPUFXNMERHUYB746PA72NZDA7LTJU3CHSRGV6EYPWDQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.436699445 (166h33m28.575683639s) [noise=true , address=145.79.227.38:4978 , piece_id=DZ3YISF75444FUUJGNDVG3SPZSWOIRJEVZYAKTCG73D7KIYXOXRQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.438093540 (166h33m28.577077734s) [noise=true , address=145.79.227.26:4653 , piece_id=NR3KMDIWN4WZM7ASUFHNBJJZPOI5IZJJLWTEZO6IYMRYMA4IO2TQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.439307297 (166h33m28.578291481s) [noise=true , address=145.79.227.2:4114 , piece_id=RRA45YDH5ZDU4CH5CTNQMKWPFENHYDEGE7V3NKLJ3LRXGFQORL5Q, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.439761746 (166h33m28.57874594s) [noise=true , address=145.79.227.40:4049 , piece_id=NPRJTV5UJRTE52IDTVHMFXJD7CYR2I5C2ZFLUJVAVD6ZUZMVZNAQ, seg_pos={0 0}, upload=146474]: Uploading piece...
Feb 24 18:42:09.518930921 (166h33m28.657915115s) [upload=146474]: Next returned nil segment
Feb 24 18:42:09.518940428 (166h33m28.657924612s) [upload=146474]: Waiting for error group managing segments...
Feb 24 18:42:09.528008379 (166h33m28.666992563s) [noise=true , address=145.79.227.15:4875 , piece_id=FXYLYOG4Y2MQ5NKOYUVPKZU7NOIECZGBAQT5B5WTYGHJ7TQ54C4A, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.528221628 (166h33m28.667205812s) [noise=true , address=145.79.227.39:4018 , piece_id=3QEOPGVK32JLEJV2P4HVZY763WAZCMBKFKRKYKAKW2RUZIAUXODQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.528377569 (166h33m28.667361753s) [noise=true , address=145.79.227.49:4343 , piece_id=HJMNJUJ62JDEKJH55MG4ARTKENXN6ENSAB6SO2VXXCXF7CSGS6PQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.529310872 (166h33m28.668295066s) [noise=true , address=145.79.227.16:4914 , piece_id=LOH6G33NOTSPX2EHGYPSEXHCR2VLUWRR2BYRZO5KATQSN3KUTTZA, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.531061212 (166h33m28.670045397s) [noise=true , address=145.79.227.35:4486 , piece_id=D6SHML7OT3YNTWYLFOQWBDLUZCBOE3HIU4E7N2F7SLGOGCUQ5RMQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.533467337 (166h33m28.672451532s) [noise=true , address=145.79.227.11:4731 , piece_id=MX5CYRWIYNI55NDKVMWGA3AFS2LMUPUU34246CNIBA24BWDVARIQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.540788244 (166h33m28.679772438s) [noise=true , address=145.79.227.26:4653 , piece_id=NR3KMDIWN4WZM7ASUFHNBJJZPOI5IZJJLWTEZO6IYMRYMA4IO2TQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.541335998 (166h33m28.680320192s) [noise=true , address=145.79.227.20:4443 , piece_id=MYRORKSVAKD76XH6EJ7NVP5P5YNQ7TLESLF2LAA22B6PH5SHTQCQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.542170426 (166h33m28.68115462s) [noise=true , address=145.79.227.40:4049 , piece_id=NPRJTV5UJRTE52IDTVHMFXJD7CYR2I5C2ZFLUJVAVD6ZUZMVZNAQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.543683784 (166h33m28.682667968s) [noise=true , address=77.90.31.18:4805 , piece_id=6VL6GAR26VPUFXNMERHUYB746PA72NZDA7LTJU3CHSRGV6EYPWDQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.545376767 (166h33m28.684360961s) [noise=true , address=45.91.116.5:4297 , piece_id=WZJLLLQIUTPKW247TATRLH5K425MX2ZD76JNGJQT4KME7AHV2FFA, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.546176590 (166h33m28.685160784s) [noise=true , address=145.79.227.38:4978 , piece_id=DZ3YISF75444FUUJGNDVG3SPZSWOIRJEVZYAKTCG73D7KIYXOXRQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.549820388 (166h33m28.688804572s) [noise=true , address=145.79.227.2:4114 , piece_id=RRA45YDH5ZDU4CH5CTNQMKWPFENHYDEGE7V3NKLJ3LRXGFQORL5Q, seg_pos={0 0}, upload=146474]: Done uploading piece. err: <nil>
Feb 24 18:42:09.549827611 (166h33m28.688811795s) [seg_pos={0 0}, upload=146474]: Satisfied cohort requirements after 13 pieces
Feb 24 18:42:09.549977892 (166h33m28.688962086s) [noise=true , address=145.79.227.41:4088 , piece_id=S25UWWCAVJUXYEGMU45PHVTQX4L5BRFAKEIHDQD426UYI5XOEZIA, seg_pos={0 0}, upload=146474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj): protocol: expected piece hash; context canceled; EOF
Feb 24 18:42:09.549983733 (166h33m28.688967917s) [noise=true , address=145.79.227.41:4088 , piece_id=S25UWWCAVJUXYEGMU45PHVTQX4L5BRFAKEIHDQD426UYI5XOEZIA, seg_pos={0 0}, upload=146474]: Long tail cancellation due to optimal threshold
Feb 24 18:42:09.550007498 (166h33m28.688991692s) [noise=true , address=145.79.227.43:4157 , piece_id=WM3UGYNXPAYIWMZT3DXFLV44LWGMAKHN7YUWG4A2HOO26HSPCGKQ, seg_pos={0 0}, upload=146474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF
Feb 24 18:42:09.550013860 (166h33m28.688998044s) [noise=true , address=145.79.227.43:4157 , piece_id=WM3UGYNXPAYIWMZT3DXFLV44LWGMAKHN7YUWG4A2HOO26HSPCGKQ, seg_pos={0 0}, upload=146474]: Long tail cancellation due to optimal threshold
Feb 24 18:42:09.550076105 (166h33m28.689060299s) [noise=true , address=145.79.227.37:4567 , piece_id=KLDMATQWJ5BJBN6EH76Q27YKWK6MFT74AAR543RYW2NX6YBHCO7Q, seg_pos={0 0}, upload=146474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF
Feb 24 18:42:09.550081025 (166h33m28.689065219s) [noise=true , address=145.79.227.37:4567 , piece_id=KLDMATQWJ5BJBN6EH76Q27YKWK6MFT74AAR543RYW2NX6YBHCO7Q, seg_pos={0 0}, upload=146474]: Long tail cancellation due to optimal threshold
Feb 24 18:42:09.550112364 (166h33m28.689096558s) [seg_pos={0 0}, upload=146474]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 24 18:42:09.550210447 (166h33m28.689194631s) [upload=146474]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 24 18:42:09.550214775 (166h33m28.689198959s) [upload=146474]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 24 18:43:51.977860968 (166h35m11.116845182s) [upload=146474]: Done uploading segments...
Feb 24 18:43:51.977945004 (166h35m11.116929219s) [upload=146474]: Upload finished. err: metaclient: context deadline exceeded
Duration: 529.525239ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
30 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX........] duration:
662.509094ms trace:
https://jaeger.dev.storj.io/trace/51430d55a27ab7ca
FAIL (2026-02-24 18:48:47 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 662.52864ms
core
uplink
us-select-1
us1-placement-20
|