Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
11 out of 60 is failed (threshold=3) [.............................X....XX.........XX.XX..XXX....X] duration:
30.000548187s
FAIL (2025-08-26 05:20:03 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000562323s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
14 out of 60 is failed (threshold=3) [......................XX.......X..X.......XXXXXXXX.XX.......] duration:
22.084599871s
FAIL (2025-08-26 05:20:00 UTC)
operation: Commit uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 22.084614529s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............................................X.X..X.......] duration:
34.910659769s
FAIL (2025-08-26 05:19:44 UTC)
orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 185F3927EAB52717, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled)
Duration: 34.910679647s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
5 out of 60 is failed (threshold=3) [.............................................X.....XX.XX....] duration:
8.036883991s remote:
"tcp:45.38.24.2:443" response headers:
Accept-Ranges=bytes Content-Type=application/xml Server=Storj Vary=Origin X-Amz-Request-Id=185F394AB595C12F X-Request-Id=N1znMTMT8Va X-Xss-Protection=1; mode=block Content-Length=433 Content-Security-Policy=block-all-mixed-content Date=Tue, 26 Aug 2025 05:16:13 GMT target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-26 05:19:39 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis eval failed: dial tcp 10.37.0.83:6379: operation was canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>185F394AB595C12F</RequestId><HostId></HostId></Error>"
Duration: 8.036905641s
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.......................XX.......X..X.......XXXXXXXX.XX......] duration:
29.155960918s
FAIL (2025-08-26 05:19:38 UTC)
operation: Commit uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 29.15598377s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
5 out of 60 is failed (threshold=3) [...............................................X.X..XX.....X] duration:
3.987679739s
FAIL (2025-08-26 05:19:37 UTC)
operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled ======LOGS====== Aug 26 05:19:33.789655908 (13h15m49.014838985s) [upload=11650]: Starting upload Aug 26 05:19:33.789738403 (13h15m49.01492148s) [upload=11650]: Done starting upload Aug 26 05:19:33.789799137 (13h15m49.014982224s) [upload=11650]: Uploading segments... Aug 26 05:19:33.789882654 (13h15m49.01506573s) [upload=11650]: Got next segment. Inline: true Aug 26 05:19:33.789891009 (13h15m49.015074086s) [upload=11650]: Waiting for error group managing segments... Aug 26 05:19:33.789942085 (13h15m49.015125162s) [upload=11650]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 26 05:19:33.789947546 (13h15m49.015130624s) [upload=11650]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 26 05:19:33.789951443 (13h15m49.015134521s) [upload=11650]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 26 05:19:37.763117992 (13h15m52.988301059s) [upload=11650]: Done uploading segments... Aug 26 05:19:37.763136176 (13h15m52.988319243s) [upload=11650]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 3.987696149s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
4 out of 60 is failed (threshold=3) [................................................X.X..XX.....] duration:
43.073286879s
FAIL (2025-08-26 05:19:33 UTC)
operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled ======LOGS====== Aug 26 05:13:43.987058282 (13h9m59.212241339s) [upload=11567]: Starting upload Aug 26 05:13:43.987144814 (13h9m59.212327871s) [upload=11567]: Done starting upload Aug 26 05:13:43.987212292 (13h9m59.212395349s) [upload=11567]: Uploading segments... Aug 26 05:13:43.987257306 (13h9m59.212440363s) [upload=11567]: Got next segment. Inline: true Aug 26 05:13:43.987263478 (13h9m59.212446525s) [upload=11567]: Waiting for error group managing segments... Aug 26 05:13:43.987304054 (13h9m59.212487152s) [upload=11567]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 26 05:13:43.987309574 (13h9m59.212492631s) [upload=11567]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 26 05:13:43.987313713 (13h9m59.21249677s) [upload=11567]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 26 05:14:09.912697999 (13h10m25.137881087s) [upload=11567]: Done uploading segments... Aug 26 05:14:09.912726052 (13h10m25.137909209s) [upload=11567]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 43.073312576s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
9 out of 60 is failed (threshold=3) [...........................XX.......X..........XX.X.X.XX....] duration:
20.160582337s
FAIL (2025-08-26 05:19:28 UTC)
operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled ======LOGS====== Aug 26 05:14:29.029036501 (13h10m44.254219628s) [upload=11579]: Starting upload Aug 26 05:14:29.029135167 (13h10m44.254318244s) [upload=11579]: Done starting upload Aug 26 05:14:29.029207823 (13h10m44.25439091s) [upload=11579]: Uploading segments... Aug 26 05:14:29.029253299 (13h10m44.254436376s) [upload=11579]: Got next segment. Inline: true Aug 26 05:14:29.029259290 (13h10m44.254442367s) [upload=11579]: Waiting for error group managing segments... Aug 26 05:14:29.029307520 (13h10m44.254490598s) [upload=11579]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 26 05:14:29.029313171 (13h10m44.254496248s) [upload=11579]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 26 05:14:29.029317299 (13h10m44.254500376s) [upload=11579]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 26 05:14:41.554926683 (13h10m56.78010974s) [upload=11579]: Done uploading segments... Aug 26 05:14:41.554940789 (13h10m56.780123846s) [upload=11579]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 20.160607354s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [....................................X..........X....X.......] duration:
8.042057837s remote:
"tcp:136.0.77.2:443" response headers:
X-Request-Id=VcCcTMkH9av Content-Length=423 Content-Type=application/xml Vary=Origin X-Amz-Request-Id=185F392FFCED1105 X-Xss-Protection=1; mode=block Date=Tue, 26 Aug 2025 05:14:18 GMT Accept-Ranges=bytes Content-Security-Policy=block-all-mixed-content Server=Storj target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-26 05:19:23 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis eval failed: dial tcp 10.37.0.83:6379: operation was canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>185F392FFCED1105</RequestId><HostId></HostId></Error>"
Duration: 8.042107841s
edge
gateway
vhost
bgp
us1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
4 out of 60 is failed (threshold=6) [................................X...............X..X....X...] duration:
24.30951564s remote:
"tcp:136.0.77.2:443" response headers:
Access-Control-Allow-Methods=GET, HEAD Access-Control-Allow-Origin=* Date=Tue, 26 Aug 2025 05:15:54 GMT Accept-Ranges=bytes X-Request-Id=PzQG5qDgV69 Content-Type=text/html; charset=utf-8 Access-Control-Allow-Headers=* target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt
FAIL (2025-08-26 05:19:19 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<!DOCTYPE html>\n<html lang=\"en\">\n\n<head>\n <meta charset=\"utf-8\">\n <title>Error | Storj</title>\n\n <meta property=\"og:url\" content=\"https://link.storjshare.io\" />\n <meta property=\"og:title\" content=\"Error | Storj\" />\n <meta property=\"og:description\" name=\"description\" content=\"Shared content - Storj\" />\n \n <meta property=\"og:image\" content=\"https://link.storjshare.io/static/img/file.png?v=f0d1e95\" />\n \n\n <meta name=\"twitter:card\" content=\"summary\" />\n <meta name=\"twitter:site\" content=\"@storj\" />\n <meta name=\"twitter:title\" content=\"Error | Storj\" />\n <meta name=\"twitter:description\" content=\"Shared content - Storj\" />\n \n <meta name=\"twitter:image\" content=\"https://link.storjshare.io/static/img/file.png?v=f0d1e95\" />\n \n\n <link rel=\"shortcut icon\"\n href=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAIAAAD8GO2jAAAC40lEQVRIDe1US0iUURQ+V8fJmcnykTJmNA0tMszETUVlCrkoR+2lWdFDk7AHlNQiCGaREhlUli3ETSRFqx4UlAi5ySgqS8hQiXLMTKca35k6j3O6d5zE//rnuNCF4Mdd3HvO+c53/3P/cxgRwUwiY"
Duration: 24.309548532s
edge
linksharing
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [...........................X..................XXX....XXX....] duration:
20.396641219s
FAIL (2025-08-26 05:19:17 UTC)
orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 429, RequestID: 185F393D8A8A6220, HostID: , api error SlowDown: Please reduce your request rate.
Duration: 20.396661706s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..........................X.....X...........X.X.XX.XXXXX....] duration:
20.346025539s
FAIL (2025-08-26 05:19:10 UTC)
operation: DownloadObject
======ERROR======
uplink: metaclient: accounting backend: Redis eval failed: dial tcp 10.37.0.83:6379: operation was canceled
======LOGS======
Aug 26 05:15:11.856866547 (13h11m27.082049594s) [upload=11590]: Starting upload
Aug 26 05:15:11.856947388 (13h11m27.082130445s) [upload=11590]: Done starting upload
Aug 26 05:15:11.857007762 (13h11m27.082190808s) [upload=11590]: Uploading segments...
Aug 26 05:15:11.857250268 (13h11m27.082433315s) [upload=11590]: Got next segment. Inline: false
Aug 26 05:15:11.857286635 (13h11m27.082469692s) [upload=11590]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 26 05:15:11.857293298 (13h11m27.082476355s) [upload=11590]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 26 05:15:22.858314105 (13h11m38.083497183s) [seg_pos={0 0}, upload=11590]: Begin upload segment...
Aug 26 05:15:22.858367536 (13h11m38.083550623s) [seg_pos={0 0}, upload=11590]: Done begin upload segment.
Aug 26 05:15:22.858400479 (13h11m38.083583556s) [noise=true , address=45.91.116.14:4190 , piece_id=4R7WJUCTCWBXGOWT26AKRAJJJGVAWVOQWQDI5CUAXWAGKBO3LUSA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.858456674 (13h11m38.083639761s) [noise=true , address=77.90.31.15:4699 , piece_id=DK3JB4BSU5D4DGRDKIGEYH4J2XFGGHQGSYLDLG76OIILI5QQF44Q, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.858923801 (13h11m38.084106879s) [noise=true , address=45.91.116.5:4298 , piece_id=XABSOODUDZQAMVECL6IVUVWMQFZX3IAOZU7D4UHF3KIHP3B3CMPA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.859079042 (13h11m38.08426213s) [noise=true , address=45.91.116.9:4261 , piece_id=PK26MWMBLSKNWB2PMTGGAF6NZ4EAY5BIUQP3CVGYOXRXI2NEWK4Q, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.859314645 (13h11m38.084497722s) [noise=true , address=45.91.116.13:4155 , piece_id=2ZPF5WCMSCW3S2I2RC2EVJIDNFKH775TIE3J7Q3AJRHR5JJKBJ3A, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.859514331 (13h11m38.084697417s) [noise=true , address=45.91.116.13:4154 , piece_id=UVDYGWF7HRYICI4YDHO2GEBAG5YJX24MFKIVGIVJMFAGKH6IQIOQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.859739903 (13h11m38.084922981s) [noise=true , address=45.91.116.11:4118 , piece_id=XADWBNSBO7PIOWZROEFOVQVYZBY7DXARHZOYDIOLBA54ZPFU4NJQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860027705 (13h11m38.085210792s) [noise=true , address=104.36.82.7:4909 , piece_id=7YOQSRLJITDS42AURAM4BP24GSE42OPR4JCNLQ6GD6Y6M5ZZLMIQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860244512 (13h11m38.085427589s) [noise=true , address=77.90.31.13:4626 , piece_id=6OVHSO55T4UTUWVLDB4WT2QLK5U3JTIDI3ACVPYZB4LUHTBOZ3EQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860413158 (13h11m38.085596236s) [noise=true , address=45.91.116.9:4263 , piece_id=YPCUKYKUKEDWDZFRYMNCULCMUAOVLYA3DTZJU7KRFUGPQCE5Z6ZA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860662036 (13h11m38.085845133s) [noise=true , address=104.36.82.12:4150 , piece_id=Y4AJKPQYW5A2PSV5F6IZ27C6XE2BYU7F7OUWGNAQXDKCZPOTGDHA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860962371 (13h11m38.086145457s) [noise=true , address=77.90.31.21:4879 , piece_id=4I3TJN7NUHC2QOJ6JLLSXTM6SQ7UMEQCWYBIRNYNRAJYKENJBQQQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.861160522 (13h11m38.0863436s) [noise=true , address=104.36.82.2:4729 , piece_id=YBU6CLRMUTMBQJMBTQ7J4YBXPDGDSAIKTY36HWSGBXOXTY6UPKGA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.861340972 (13h11m38.086524019s) [noise=true , address=77.90.31.14:4663 , piece_id=RM3L6KVSXK7XPWIS2PB63PBSKC4XF22EKHY3PAJE264NNG65TRHQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.861443835 (13h11m38.086626881s) [noise=true , address=45.91.116.4:4939 , piece_id=FKDXX5QA7VPHPQ5WW4W3OJCKBFISFTGZHQSOPCF2LKUZSFOJHLHA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.861699475 (13h11m38.086882522s) [noise=true , address=77.90.31.11:4554 , piece_id=DXNENQZMA57JJ4QB46OSDDVN6RRTTT6DAWNJ3JTUWWHYL2XUA7GA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.990256495 (13h11m38.215439542s) [upload=11590]: Next returned nil segment
Aug 26 05:15:22.990276573 (13h11m38.21545963s) [upload=11590]: Waiting for error group managing segments...
Aug 26 05:15:22.998364304 (13h11m38.223547351s) [noise=true , address=45.91.116.9:4261 , piece_id=PK26MWMBLSKNWB2PMTGGAF6NZ4EAY5BIUQP3CVGYOXRXI2NEWK4Q, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.998564650 (13h11m38.223747697s) [noise=true , address=45.91.116.14:4190 , piece_id=4R7WJUCTCWBXGOWT26AKRAJJJGVAWVOQWQDI5CUAXWAGKBO3LUSA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.998988045 (13h11m38.224171092s) [noise=true , address=45.91.116.11:4118 , piece_id=XADWBNSBO7PIOWZROEFOVQVYZBY7DXARHZOYDIOLBA54ZPFU4NJQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.999143488 (13h11m38.224326535s) [noise=true , address=45.91.116.9:4263 , piece_id=YPCUKYKUKEDWDZFRYMNCULCMUAOVLYA3DTZJU7KRFUGPQCE5Z6ZA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.999571281 (13h11m38.224754328s) [noise=true , address=77.90.31.13:4626 , piece_id=6OVHSO55T4UTUWVLDB4WT2QLK5U3JTIDI3ACVPYZB4LUHTBOZ3EQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.999598783 (13h11m38.22478183s) [noise=true , address=45.91.116.13:4154 , piece_id=UVDYGWF7HRYICI4YDHO2GEBAG5YJX24MFKIVGIVJMFAGKH6IQIOQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.000104903 (13h11m38.22528795s) [noise=true , address=45.91.116.5:4298 , piece_id=XABSOODUDZQAMVECL6IVUVWMQFZX3IAOZU7D4UHF3KIHP3B3CMPA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.000133807 (13h11m38.225316864s) [noise=true , address=77.90.31.21:4879 , piece_id=4I3TJN7NUHC2QOJ6JLLSXTM6SQ7UMEQCWYBIRNYNRAJYKENJBQQQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.000584494 (13h11m38.225767551s) [noise=true , address=45.91.116.13:4155 , piece_id=2ZPF5WCMSCW3S2I2RC2EVJIDNFKH775TIE3J7Q3AJRHR5JJKBJ3A, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.000818082 (13h11m38.226001139s) [noise=true , address=45.91.116.4:4939 , piece_id=FKDXX5QA7VPHPQ5WW4W3OJCKBFISFTGZHQSOPCF2LKUZSFOJHLHA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.001007669 (13h11m38.226190736s) [noise=true , address=104.36.82.7:4909 , piece_id=7YOQSRLJITDS42AURAM4BP24GSE42OPR4JCNLQ6GD6Y6M5ZZLMIQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.001806589 (13h11m38.226989646s) [noise=true , address=77.90.31.14:4663 , piece_id=RM3L6KVSXK7XPWIS2PB63PBSKC4XF22EKHY3PAJE264NNG65TRHQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.001962061 (13h11m38.227145108s) [noise=true , address=77.90.31.15:4699 , piece_id=DK3JB4BSU5D4DGRDKIGEYH4J2XFGGHQGSYLDLG76OIILI5QQF44Q, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.001984242 (13h11m38.22716732s) [seg_pos={0 0}, upload=11590]: Segment reached optimal threshold of 13 pieces.
Aug 26 05:15:23.002128384 (13h11m38.227311431s) [noise=true , address=104.36.82.12:4150 , piece_id=Y4AJKPQYW5A2PSV5F6IZ27C6XE2BYU7F7OUWGNAQXDKCZPOTGDHA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF
Aug 26 05:15:23.002141278 (13h11m38.227324335s) [noise=true , address=77.90.31.11:4554 , piece_id=DXNENQZMA57JJ4QB46OSDDVN6RRTTT6DAWNJ3JTUWWHYL2XUA7GA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF
Aug 26 05:15:23.002152920 (13h11m38.227335967s) [noise=true , address=104.36.82.2:4729 , piece_id=YBU6CLRMUTMBQJMBTQ7J4YBXPDGDSAIKTY36HWSGBXOXTY6UPKGA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF
Aug 26 05:15:23.002175222 (13h11m38.227358269s) [seg_pos={0 0}, upload=11590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 26 05:15:23.002254370 (13h11m38.227437437s) [upload=11590]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 26 05:15:23.002260071 (13h11m38.227443118s) [upload=11590]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 26 05:15:31.060086659 (13h11m46.285269727s) [upload=11590]: Done uploading segments...
Aug 26 05:15:31.060104984 (13h11m46.285288061s) [upload=11590]: Upload finished. err: <nil>
Duration: 20.346045065s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [.....................................X..........X....X......] duration:
19.727027625s remote:
"tcp:136.0.77.2:443" response headers:
X-Request-Id=VcCcTMkH9av Content-Length=423 Content-Type=application/xml Vary=Origin X-Amz-Request-Id=185F392FFCED1105 X-Xss-Protection=1; mode=block Date=Tue, 26 Aug 2025 05:14:18 GMT Accept-Ranges=bytes Content-Security-Policy=block-all-mixed-content Server=Storj target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-26 05:19:08 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis eval failed: dial tcp 10.37.0.83:6379: operation was canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>185F392FFCED1105</RequestId><HostId></HostId></Error>"
Duration: 19.727048584s
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [......................................X........XX.X.X.X.....] duration:
4.014586039s
FAIL (2025-08-26 05:19:04 UTC)
operation: Commit
======ERROR======
uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
======LOGS======
0704832 (13h11m2.905887889s) [upload=11582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 26 05:14:47.902776638 (13h11m3.127959686s) [seg_pos={0 0}, upload=11582]: Begin upload segment...
Aug 26 05:14:47.902820661 (13h11m3.128003709s) [seg_pos={0 0}, upload=11582]: Done begin upload segment.
Aug 26 05:14:47.902849054 (13h11m3.128032111s) [noise=true , address=45.91.116.11:4146 , piece_id=HIWPDZR3CTF22YZCYH73QMJFSGET7QTBHRMNOLVMTSU6RJXSWTFA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.902873490 (13h11m3.128056537s) [noise=true , address=77.90.31.14:4694 , piece_id=O6K4VOHGODJGHAKXAGEKEAXU7ZXHIEVTE2S6KBKWSUJVZKACUWNQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903189514 (13h11m3.128372571s) [noise=true , address=104.36.82.13:4214 , piece_id=MBC7UIGS4Y7LUYQKQTG33CMKX7O25ANAV73ONFHCWKX7MLDIXOVA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903342101 (13h11m3.128525148s) [noise=true , address=45.91.116.14:4203 , piece_id=C4YPVPOP56QR7P6RLTPZYXDBOJGNVSI7QS52DQRKESU6MBUEXN2A, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903413926 (13h11m3.128596973s) [noise=true , address=104.36.82.50:4030 , piece_id=FIQJULHAAWQQXVV3BTH5XIIYNUIVLOPXVEEOJWETTY6KBWKCX2WQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903641763 (13h11m3.12882482s) [noise=true , address=79.127.205.246:4268 , piece_id=WMRJ2R3PWW4PEJKMI4WO6RDGV24VBMFDJMNA5ST4XXWNG3EZGYYA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903697358 (13h11m3.128880406s) [noise=true , address=104.36.82.44:4873 , piece_id=CVFBS5TPDC7DSBY2MGZH4EL5DO3PLJDE2GJCZ2TL6JV7ETU2E34A, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.904382555 (13h11m3.129565612s) [noise=true , address=77.90.31.19:4855 , piece_id=V36EQSYCAE3O2GFK4KNVLKSYOJADKRIOMRBVSOHRI3RNL3AFEYNA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.904391010 (13h11m3.129574068s) [noise=true , address=104.36.82.6:4891 , piece_id=HBARHJPGMLXHPEEA27KEMNMJVRFWTWR74YQL5XTO7KYRZ266VJXA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.904787035 (13h11m3.129970082s) [noise=true , address=104.36.82.60:4354 , piece_id=HFCYUWFHQ6AJA7REULM6KRFPMDFBBKRRJ22VQXSVTM5S4X3BX3PA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.905089312 (13h11m3.130272359s) [noise=true , address=104.36.82.49:4987 , piece_id=ECBZZ23GJA5TWTPB2X7GZBFXIY5VL5FJR77KBF64QCO5BEVUALCQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.905389606 (13h11m3.130572683s) [noise=true , address=104.36.82.10:4085 , piece_id=AZBDQYQUON4UVMJ3F4ULQXTNBB2OSWFFC72KU3MA2J7RGBJZELWA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.905880078 (13h11m3.131063125s) [noise=true , address=79.127.201.215:4349 , piece_id=BU6KUUSE77HTCME4DFHPPMADIFAFHI43PL2JPDEHZHZEYJMRXR4Q, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.906031132 (13h11m3.131214179s) [noise=true , address=45.91.116.7:4030 , piece_id=JX6NWRZODJ3PEDOU43U5KQXHYBFYBPYJNDISY2T3DLK5NYZ5I7GA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.906575734 (13h11m3.131758791s) [noise=true , address=79.127.205.245:4244 , piece_id=G25ULKCGLHJCT7QCOP6ZSPXA7CHOS3KNIRZMRQT6NWRZPVRTCZCQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.906838247 (13h11m3.132021314s) [noise=true , address=45.91.116.6:4992 , piece_id=6G7PY5YY32FTZBPRVLKSHSUH7KNYRIGHAFOPT7AVBCBM7HYH3MVQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.907199135 (13h11m3.132382192s) [noise=true , address=45.91.116.13:4169 , piece_id=7ND7C76WDTB77MRO3RPSQDGQFOK7H5WOBSD7RCVI7OUA5H4HSSXA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.910863127 (13h11m3.136046194s) [noise=true , address=104.36.82.23:4456 , piece_id=NOIRLI7AVEW4P3BRPJI27WOSFOO2WP2E6RDPFWWGHEWUKZ7HTF7Q, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.996930147 (13h11m3.222113204s) [upload=11582]: Next returned nil segment
Aug 26 05:14:47.996953811 (13h11m3.222136868s) [upload=11582]: Waiting for error group managing segments...
Aug 26 05:14:48.004359973 (13h11m3.229543019s) [noise=true , address=45.91.116.7:4030 , piece_id=JX6NWRZODJ3PEDOU43U5KQXHYBFYBPYJNDISY2T3DLK5NYZ5I7GA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.005508650 (13h11m3.230691697s) [noise=true , address=45.91.116.11:4146 , piece_id=HIWPDZR3CTF22YZCYH73QMJFSGET7QTBHRMNOLVMTSU6RJXSWTFA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.005522095 (13h11m3.230705142s) [noise=true , address=45.91.116.14:4203 , piece_id=C4YPVPOP56QR7P6RLTPZYXDBOJGNVSI7QS52DQRKESU6MBUEXN2A, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.006008057 (13h11m3.231191105s) [noise=true , address=45.91.116.13:4169 , piece_id=7ND7C76WDTB77MRO3RPSQDGQFOK7H5WOBSD7RCVI7OUA5H4HSSXA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.008257333 (13h11m3.23344037s) [noise=true , address=104.36.82.49:4987 , piece_id=ECBZZ23GJA5TWTPB2X7GZBFXIY5VL5FJR77KBF64QCO5BEVUALCQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.008662254 (13h11m3.233845311s) [noise=true , address=104.36.82.44:4873 , piece_id=CVFBS5TPDC7DSBY2MGZH4EL5DO3PLJDE2GJCZ2TL6JV7ETU2E34A, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.009403165 (13h11m3.234586212s) [noise=true , address=77.90.31.14:4694 , piece_id=O6K4VOHGODJGHAKXAGEKEAXU7ZXHIEVTE2S6KBKWSUJVZKACUWNQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.009804760 (13h11m3.234987806s) [noise=true , address=77.90.31.19:4855 , piece_id=V36EQSYCAE3O2GFK4KNVLKSYOJADKRIOMRBVSOHRI3RNL3AFEYNA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.011645477 (13h11m3.236828524s) [noise=true , address=104.36.82.13:4214 , piece_id=MBC7UIGS4Y7LUYQKQTG33CMKX7O25ANAV73ONFHCWKX7MLDIXOVA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.016428189 (13h11m3.241611427s) [noise=true , address=104.36.82.50:4030 , piece_id=FIQJULHAAWQQXVV3BTH5XIIYNUIVLOPXVEEOJWETTY6KBWKCX2WQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.016598520 (13h11m3.241781567s) [noise=true , address=104.36.82.10:4085 , piece_id=AZBDQYQUON4UVMJ3F4ULQXTNBB2OSWFFC72KU3MA2J7RGBJZELWA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.017570516 (13h11m3.242753553s) [noise=true , address=104.36.82.6:4891 , piece_id=HBARHJPGMLXHPEEA27KEMNMJVRFWTWR74YQL5XTO7KYRZ266VJXA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.030796789 (13h11m3.255979836s) [noise=true , address=104.36.82.60:4354 , piece_id=HFCYUWFHQ6AJA7REULM6KRFPMDFBBKRRJ22VQXSVTM5S4X3BX3PA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.089222399 (13h11m3.314405456s) [noise=true , address=104.36.82.23:4456 , piece_id=NOIRLI7AVEW4P3BRPJI27WOSFOO2WP2E6RDPFWWGHEWUKZ7HTF7Q, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.089247306 (13h11m3.314430354s) [seg_pos={0 0}, upload=11582]: Segment reached optimal threshold of 14 pieces.
Aug 26 05:14:48.089472019 (13h11m3.314655076s) [noise=true , address=45.91.116.6:4992 , piece_id=6G7PY5YY32FTZBPRVLKSHSUH7KNYRIGHAFOPT7AVBCBM7HYH3MVQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx): protocol: expected piece hash; context canceled; EOF
Aug 26 05:14:48.089527372 (13h11m3.314710429s) [noise=true , address=79.127.201.215:4349 , piece_id=BU6KUUSE77HTCME4DFHPPMADIFAFHI43PL2JPDEHZHZEYJMRXR4Q, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y1KYQJsvbNxfySU7mXySUmJtEDVAE7fst1XVRHU5N8Txmjn5T): protocol: write tcp 5.161.89.252:45582->79.127.201.215:4349: use of closed network connection; write tcp 5.161.89.252:45582->79.127.201.215:4349: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:45582->79.127.201.215:4349: use of closed network connection
Aug 26 05:14:48.089730384 (13h11m3.314913441s) [noise=true , address=79.127.205.246:4268 , piece_id=WMRJ2R3PWW4PEJKMI4WO6RDGV24VBMFDJMNA5ST4XXWNG3EZGYYA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k72iHWdcnM2JKs7VPTQFQwmpQ835HmAP8KpXxjMbEy5QxfzG): protocol: read tcp 5.161.89.252:10486->79.127.205.246:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 26 05:14:48.089765280 (13h11m3.314948327s) [noise=true , address=79.127.205.245:4244 , piece_id=G25ULKCGLHJCT7QCOP6ZSPXA7CHOS3KNIRZMRQT6NWRZPVRTCZCQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JFy3MxMFZYnjxJmujGpmcTdpVz6qLUMKf1aB3eYW236kCYgng): protocol: read tcp 5.161.89.252:25038->79.127.205.245:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 26 05:14:48.089773956 (13h11m3.314957003s) [seg_pos={0 0}, upload=11582]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 26 05:14:48.089836283 (13h11m3.31501934s) [upload=11582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 26 05:14:48.089842094 (13h11m3.315025141s) [upload=11582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 26 05:14:48.256048660 (13h11m3.481231707s) [upload=11582]: Done uploading segments...
Aug 26 05:14:48.256071674 (13h11m3.481254731s) [upload=11582]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 4.014616006s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
9 out of 60 is failed (threshold=3) [.........................X.....XX..........XX....X..X.XX....] duration:
12.455999575s
FAIL (2025-08-26 05:19:03 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 12.456028119s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [................................................X.X..X......] duration:
46.811406996s
FAIL (2025-08-26 05:19:02 UTC)
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 185F3927EAB52717, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled)
Duration: 46.811425541s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [..............................X....XX.........XX.XX..XXX....] duration:
20.196967274s
FAIL (2025-08-26 05:19:00 UTC)
orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 185F3945025C03C0, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled)
Duration: 20.197039439s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
4 out of 60 is failed (threshold=6) [.................................X...............X..X....X..] duration:
36.99295043s remote:
"tcp:136.0.77.2:443" response headers:
Access-Control-Allow-Methods=GET, HEAD Access-Control-Allow-Origin=* Date=Tue, 26 Aug 2025 05:15:54 GMT Accept-Ranges=bytes X-Request-Id=PzQG5qDgV69 Content-Type=text/html; charset=utf-8 Access-Control-Allow-Headers=* target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt
FAIL (2025-08-26 05:18:54 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<!DOCTYPE html>\n<html lang=\"en\">\n\n<head>\n <meta charset=\"utf-8\">\n <title>Error | Storj</title>\n\n <meta property=\"og:url\" content=\"https://link.storjshare.io\" />\n <meta property=\"og:title\" content=\"Error | Storj\" />\n <meta property=\"og:description\" name=\"description\" content=\"Shared content - Storj\" />\n \n <meta property=\"og:image\" content=\"https://link.storjshare.io/static/img/file.png?v=f0d1e95\" />\n \n\n <meta name=\"twitter:card\" content=\"summary\" />\n <meta name=\"twitter:site\" content=\"@storj\" />\n <meta name=\"twitter:title\" content=\"Error | Storj\" />\n <meta name=\"twitter:description\" content=\"Shared content - Storj\" />\n \n <meta name=\"twitter:image\" content=\"https://link.storjshare.io/static/img/file.png?v=f0d1e95\" />\n \n\n <link rel=\"shortcut icon\"\n href=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAIAAAD8GO2jAAAC40lEQVRIDe1US0iUURQ+V8fJmcnykTJmNA0tMszETUVlCrkoR+2lWdFDk7AHlNQiCGaREhlUli3ETSRFqx4UlAi5ySgqS8hQiXLMTKca35k6j3O6d5zE//rnuNCF4Mdd3HvO+c53/3P/cxgRwUwiY"
Duration: 36.992971068s
edge
linksharing
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
5 out of 60 is failed (threshold=3) [..............................................X.....XX.XX...] duration:
8.016465905s remote:
"tcp:45.38.24.2:443" response headers:
Accept-Ranges=bytes Content-Type=application/xml Server=Storj Vary=Origin X-Amz-Request-Id=185F394AB595C12F X-Request-Id=N1znMTMT8Va X-Xss-Protection=1; mode=block Content-Length=433 Content-Security-Policy=block-all-mixed-content Date=Tue, 26 Aug 2025 05:16:13 GMT target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-26 05:18:51 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis eval failed: dial tcp 10.37.0.83:6379: operation was canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>185F394AB595C12F</RequestId><HostId></HostId></Error>"
Duration: 8.016481925s
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
9 out of 60 is failed (threshold=3) [..........................X.....XX..........XX....X..X.XX...] duration:
28.963216782s
FAIL (2025-08-26 05:18:42 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 28.963239143s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
14 out of 60 is failed (threshold=3) [........................XX.......X..X.......XXXXXXXX.XX.....] duration:
7.482041516s
FAIL (2025-08-26 05:18:34 UTC)
operation: Commit uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 7.482065342s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
9 out of 60 is failed (threshold=3) [............................XX.......X..........XX.X.X.XX...] duration:
19.835302691s
FAIL (2025-08-26 05:18:30 UTC)
operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled ======LOGS====== Aug 26 05:14:29.029036501 (13h10m44.254219628s) [upload=11579]: Starting upload Aug 26 05:14:29.029135167 (13h10m44.254318244s) [upload=11579]: Done starting upload Aug 26 05:14:29.029207823 (13h10m44.25439091s) [upload=11579]: Uploading segments... Aug 26 05:14:29.029253299 (13h10m44.254436376s) [upload=11579]: Got next segment. Inline: true Aug 26 05:14:29.029259290 (13h10m44.254442367s) [upload=11579]: Waiting for error group managing segments... Aug 26 05:14:29.029307520 (13h10m44.254490598s) [upload=11579]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 26 05:14:29.029313171 (13h10m44.254496248s) [upload=11579]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 26 05:14:29.029317299 (13h10m44.254500376s) [upload=11579]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 26 05:14:41.554926683 (13h10m56.78010974s) [upload=11579]: Done uploading segments... Aug 26 05:14:41.554940789 (13h10m56.780123846s) [upload=11579]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 19.835325503s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
5 out of 60 is failed (threshold=3) [...............................................X.....XX.XX..] duration:
8.024801475s remote:
"tcp:45.38.24.2:443" response headers:
Accept-Ranges=bytes Content-Type=application/xml Server=Storj Vary=Origin X-Amz-Request-Id=185F394AB595C12F X-Request-Id=N1znMTMT8Va X-Xss-Protection=1; mode=block Content-Length=433 Content-Security-Policy=block-all-mixed-content Date=Tue, 26 Aug 2025 05:16:13 GMT target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-26 05:18:20 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis eval failed: dial tcp 10.37.0.83:6379: operation was canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>185F394AB595C12F</RequestId><HostId></HostId></Error>"
Duration: 8.024901854s
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
4 out of 60 is failed (threshold=3) [.................................................X.X..XX....] duration:
5.90005865s
FAIL (2025-08-26 05:18:16 UTC)
operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled ======LOGS====== Aug 26 05:13:43.987058282 (13h9m59.212241339s) [upload=11567]: Starting upload Aug 26 05:13:43.987144814 (13h9m59.212327871s) [upload=11567]: Done starting upload Aug 26 05:13:43.987212292 (13h9m59.212395349s) [upload=11567]: Uploading segments... Aug 26 05:13:43.987257306 (13h9m59.212440363s) [upload=11567]: Got next segment. Inline: true Aug 26 05:13:43.987263478 (13h9m59.212446525s) [upload=11567]: Waiting for error group managing segments... Aug 26 05:13:43.987304054 (13h9m59.212487152s) [upload=11567]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 26 05:13:43.987309574 (13h9m59.212492631s) [upload=11567]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 26 05:13:43.987313713 (13h9m59.21249677s) [upload=11567]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 26 05:14:09.912697999 (13h10m25.137881087s) [upload=11567]: Done uploading segments... Aug 26 05:14:09.912726052 (13h10m25.137909209s) [upload=11567]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 5.900080851s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...........................X.....X...........X.X.XX.XXXXX...] duration:
2.774094055s
FAIL (2025-08-26 05:18:09 UTC)
operation: DownloadObject
======ERROR======
uplink: metaclient: accounting backend: Redis eval failed: dial tcp 10.37.0.83:6379: operation was canceled
======LOGS======
Aug 26 05:15:11.856866547 (13h11m27.082049594s) [upload=11590]: Starting upload
Aug 26 05:15:11.856947388 (13h11m27.082130445s) [upload=11590]: Done starting upload
Aug 26 05:15:11.857007762 (13h11m27.082190808s) [upload=11590]: Uploading segments...
Aug 26 05:15:11.857250268 (13h11m27.082433315s) [upload=11590]: Got next segment. Inline: false
Aug 26 05:15:11.857286635 (13h11m27.082469692s) [upload=11590]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 26 05:15:11.857293298 (13h11m27.082476355s) [upload=11590]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 26 05:15:22.858314105 (13h11m38.083497183s) [seg_pos={0 0}, upload=11590]: Begin upload segment...
Aug 26 05:15:22.858367536 (13h11m38.083550623s) [seg_pos={0 0}, upload=11590]: Done begin upload segment.
Aug 26 05:15:22.858400479 (13h11m38.083583556s) [noise=true , address=45.91.116.14:4190 , piece_id=4R7WJUCTCWBXGOWT26AKRAJJJGVAWVOQWQDI5CUAXWAGKBO3LUSA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.858456674 (13h11m38.083639761s) [noise=true , address=77.90.31.15:4699 , piece_id=DK3JB4BSU5D4DGRDKIGEYH4J2XFGGHQGSYLDLG76OIILI5QQF44Q, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.858923801 (13h11m38.084106879s) [noise=true , address=45.91.116.5:4298 , piece_id=XABSOODUDZQAMVECL6IVUVWMQFZX3IAOZU7D4UHF3KIHP3B3CMPA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.859079042 (13h11m38.08426213s) [noise=true , address=45.91.116.9:4261 , piece_id=PK26MWMBLSKNWB2PMTGGAF6NZ4EAY5BIUQP3CVGYOXRXI2NEWK4Q, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.859314645 (13h11m38.084497722s) [noise=true , address=45.91.116.13:4155 , piece_id=2ZPF5WCMSCW3S2I2RC2EVJIDNFKH775TIE3J7Q3AJRHR5JJKBJ3A, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.859514331 (13h11m38.084697417s) [noise=true , address=45.91.116.13:4154 , piece_id=UVDYGWF7HRYICI4YDHO2GEBAG5YJX24MFKIVGIVJMFAGKH6IQIOQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.859739903 (13h11m38.084922981s) [noise=true , address=45.91.116.11:4118 , piece_id=XADWBNSBO7PIOWZROEFOVQVYZBY7DXARHZOYDIOLBA54ZPFU4NJQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860027705 (13h11m38.085210792s) [noise=true , address=104.36.82.7:4909 , piece_id=7YOQSRLJITDS42AURAM4BP24GSE42OPR4JCNLQ6GD6Y6M5ZZLMIQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860244512 (13h11m38.085427589s) [noise=true , address=77.90.31.13:4626 , piece_id=6OVHSO55T4UTUWVLDB4WT2QLK5U3JTIDI3ACVPYZB4LUHTBOZ3EQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860413158 (13h11m38.085596236s) [noise=true , address=45.91.116.9:4263 , piece_id=YPCUKYKUKEDWDZFRYMNCULCMUAOVLYA3DTZJU7KRFUGPQCE5Z6ZA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860662036 (13h11m38.085845133s) [noise=true , address=104.36.82.12:4150 , piece_id=Y4AJKPQYW5A2PSV5F6IZ27C6XE2BYU7F7OUWGNAQXDKCZPOTGDHA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860962371 (13h11m38.086145457s) [noise=true , address=77.90.31.21:4879 , piece_id=4I3TJN7NUHC2QOJ6JLLSXTM6SQ7UMEQCWYBIRNYNRAJYKENJBQQQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.861160522 (13h11m38.0863436s) [noise=true , address=104.36.82.2:4729 , piece_id=YBU6CLRMUTMBQJMBTQ7J4YBXPDGDSAIKTY36HWSGBXOXTY6UPKGA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.861340972 (13h11m38.086524019s) [noise=true , address=77.90.31.14:4663 , piece_id=RM3L6KVSXK7XPWIS2PB63PBSKC4XF22EKHY3PAJE264NNG65TRHQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.861443835 (13h11m38.086626881s) [noise=true , address=45.91.116.4:4939 , piece_id=FKDXX5QA7VPHPQ5WW4W3OJCKBFISFTGZHQSOPCF2LKUZSFOJHLHA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.861699475 (13h11m38.086882522s) [noise=true , address=77.90.31.11:4554 , piece_id=DXNENQZMA57JJ4QB46OSDDVN6RRTTT6DAWNJ3JTUWWHYL2XUA7GA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.990256495 (13h11m38.215439542s) [upload=11590]: Next returned nil segment
Aug 26 05:15:22.990276573 (13h11m38.21545963s) [upload=11590]: Waiting for error group managing segments...
Aug 26 05:15:22.998364304 (13h11m38.223547351s) [noise=true , address=45.91.116.9:4261 , piece_id=PK26MWMBLSKNWB2PMTGGAF6NZ4EAY5BIUQP3CVGYOXRXI2NEWK4Q, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.998564650 (13h11m38.223747697s) [noise=true , address=45.91.116.14:4190 , piece_id=4R7WJUCTCWBXGOWT26AKRAJJJGVAWVOQWQDI5CUAXWAGKBO3LUSA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.998988045 (13h11m38.224171092s) [noise=true , address=45.91.116.11:4118 , piece_id=XADWBNSBO7PIOWZROEFOVQVYZBY7DXARHZOYDIOLBA54ZPFU4NJQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.999143488 (13h11m38.224326535s) [noise=true , address=45.91.116.9:4263 , piece_id=YPCUKYKUKEDWDZFRYMNCULCMUAOVLYA3DTZJU7KRFUGPQCE5Z6ZA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.999571281 (13h11m38.224754328s) [noise=true , address=77.90.31.13:4626 , piece_id=6OVHSO55T4UTUWVLDB4WT2QLK5U3JTIDI3ACVPYZB4LUHTBOZ3EQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.999598783 (13h11m38.22478183s) [noise=true , address=45.91.116.13:4154 , piece_id=UVDYGWF7HRYICI4YDHO2GEBAG5YJX24MFKIVGIVJMFAGKH6IQIOQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.000104903 (13h11m38.22528795s) [noise=true , address=45.91.116.5:4298 , piece_id=XABSOODUDZQAMVECL6IVUVWMQFZX3IAOZU7D4UHF3KIHP3B3CMPA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.000133807 (13h11m38.225316864s) [noise=true , address=77.90.31.21:4879 , piece_id=4I3TJN7NUHC2QOJ6JLLSXTM6SQ7UMEQCWYBIRNYNRAJYKENJBQQQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.000584494 (13h11m38.225767551s) [noise=true , address=45.91.116.13:4155 , piece_id=2ZPF5WCMSCW3S2I2RC2EVJIDNFKH775TIE3J7Q3AJRHR5JJKBJ3A, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.000818082 (13h11m38.226001139s) [noise=true , address=45.91.116.4:4939 , piece_id=FKDXX5QA7VPHPQ5WW4W3OJCKBFISFTGZHQSOPCF2LKUZSFOJHLHA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.001007669 (13h11m38.226190736s) [noise=true , address=104.36.82.7:4909 , piece_id=7YOQSRLJITDS42AURAM4BP24GSE42OPR4JCNLQ6GD6Y6M5ZZLMIQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.001806589 (13h11m38.226989646s) [noise=true , address=77.90.31.14:4663 , piece_id=RM3L6KVSXK7XPWIS2PB63PBSKC4XF22EKHY3PAJE264NNG65TRHQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.001962061 (13h11m38.227145108s) [noise=true , address=77.90.31.15:4699 , piece_id=DK3JB4BSU5D4DGRDKIGEYH4J2XFGGHQGSYLDLG76OIILI5QQF44Q, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.001984242 (13h11m38.22716732s) [seg_pos={0 0}, upload=11590]: Segment reached optimal threshold of 13 pieces.
Aug 26 05:15:23.002128384 (13h11m38.227311431s) [noise=true , address=104.36.82.12:4150 , piece_id=Y4AJKPQYW5A2PSV5F6IZ27C6XE2BYU7F7OUWGNAQXDKCZPOTGDHA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF
Aug 26 05:15:23.002141278 (13h11m38.227324335s) [noise=true , address=77.90.31.11:4554 , piece_id=DXNENQZMA57JJ4QB46OSDDVN6RRTTT6DAWNJ3JTUWWHYL2XUA7GA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF
Aug 26 05:15:23.002152920 (13h11m38.227335967s) [noise=true , address=104.36.82.2:4729 , piece_id=YBU6CLRMUTMBQJMBTQ7J4YBXPDGDSAIKTY36HWSGBXOXTY6UPKGA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF
Aug 26 05:15:23.002175222 (13h11m38.227358269s) [seg_pos={0 0}, upload=11590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 26 05:15:23.002254370 (13h11m38.227437437s) [upload=11590]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 26 05:15:23.002260071 (13h11m38.227443118s) [upload=11590]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 26 05:15:31.060086659 (13h11m46.285269727s) [upload=11590]: Done uploading segments...
Aug 26 05:15:31.060104984 (13h11m46.285288061s) [upload=11590]: Upload finished. err: <nil>
Duration: 2.774125523s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [...............................X....XX.........XX.XX..XXX...] duration:
2.548992856s
FAIL (2025-08-26 05:18:09 UTC)
orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 185F3945025C03C0, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled)
Duration: 2.549008474s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [......................................X..........X....X.....] duration:
190.352938ms remote:
"tcp:136.0.77.2:443" response headers:
X-Request-Id=VcCcTMkH9av Content-Length=423 Content-Type=application/xml Vary=Origin X-Amz-Request-Id=185F392FFCED1105 X-Xss-Protection=1; mode=block Date=Tue, 26 Aug 2025 05:14:18 GMT Accept-Ranges=bytes Content-Security-Policy=block-all-mixed-content Server=Storj target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-26 05:18:03 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis eval failed: dial tcp 10.37.0.83:6379: operation was canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>185F392FFCED1105</RequestId><HostId></HostId></Error>"
Duration: 190.381131ms
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......................................X........XX.X.X.X....] duration:
807.001079ms
FAIL (2025-08-26 05:17:47 UTC)
operation: Commit
======ERROR======
uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
======LOGS======
0704832 (13h11m2.905887889s) [upload=11582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 26 05:14:47.902776638 (13h11m3.127959686s) [seg_pos={0 0}, upload=11582]: Begin upload segment...
Aug 26 05:14:47.902820661 (13h11m3.128003709s) [seg_pos={0 0}, upload=11582]: Done begin upload segment.
Aug 26 05:14:47.902849054 (13h11m3.128032111s) [noise=true , address=45.91.116.11:4146 , piece_id=HIWPDZR3CTF22YZCYH73QMJFSGET7QTBHRMNOLVMTSU6RJXSWTFA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.902873490 (13h11m3.128056537s) [noise=true , address=77.90.31.14:4694 , piece_id=O6K4VOHGODJGHAKXAGEKEAXU7ZXHIEVTE2S6KBKWSUJVZKACUWNQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903189514 (13h11m3.128372571s) [noise=true , address=104.36.82.13:4214 , piece_id=MBC7UIGS4Y7LUYQKQTG33CMKX7O25ANAV73ONFHCWKX7MLDIXOVA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903342101 (13h11m3.128525148s) [noise=true , address=45.91.116.14:4203 , piece_id=C4YPVPOP56QR7P6RLTPZYXDBOJGNVSI7QS52DQRKESU6MBUEXN2A, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903413926 (13h11m3.128596973s) [noise=true , address=104.36.82.50:4030 , piece_id=FIQJULHAAWQQXVV3BTH5XIIYNUIVLOPXVEEOJWETTY6KBWKCX2WQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903641763 (13h11m3.12882482s) [noise=true , address=79.127.205.246:4268 , piece_id=WMRJ2R3PWW4PEJKMI4WO6RDGV24VBMFDJMNA5ST4XXWNG3EZGYYA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903697358 (13h11m3.128880406s) [noise=true , address=104.36.82.44:4873 , piece_id=CVFBS5TPDC7DSBY2MGZH4EL5DO3PLJDE2GJCZ2TL6JV7ETU2E34A, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.904382555 (13h11m3.129565612s) [noise=true , address=77.90.31.19:4855 , piece_id=V36EQSYCAE3O2GFK4KNVLKSYOJADKRIOMRBVSOHRI3RNL3AFEYNA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.904391010 (13h11m3.129574068s) [noise=true , address=104.36.82.6:4891 , piece_id=HBARHJPGMLXHPEEA27KEMNMJVRFWTWR74YQL5XTO7KYRZ266VJXA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.904787035 (13h11m3.129970082s) [noise=true , address=104.36.82.60:4354 , piece_id=HFCYUWFHQ6AJA7REULM6KRFPMDFBBKRRJ22VQXSVTM5S4X3BX3PA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.905089312 (13h11m3.130272359s) [noise=true , address=104.36.82.49:4987 , piece_id=ECBZZ23GJA5TWTPB2X7GZBFXIY5VL5FJR77KBF64QCO5BEVUALCQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.905389606 (13h11m3.130572683s) [noise=true , address=104.36.82.10:4085 , piece_id=AZBDQYQUON4UVMJ3F4ULQXTNBB2OSWFFC72KU3MA2J7RGBJZELWA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.905880078 (13h11m3.131063125s) [noise=true , address=79.127.201.215:4349 , piece_id=BU6KUUSE77HTCME4DFHPPMADIFAFHI43PL2JPDEHZHZEYJMRXR4Q, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.906031132 (13h11m3.131214179s) [noise=true , address=45.91.116.7:4030 , piece_id=JX6NWRZODJ3PEDOU43U5KQXHYBFYBPYJNDISY2T3DLK5NYZ5I7GA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.906575734 (13h11m3.131758791s) [noise=true , address=79.127.205.245:4244 , piece_id=G25ULKCGLHJCT7QCOP6ZSPXA7CHOS3KNIRZMRQT6NWRZPVRTCZCQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.906838247 (13h11m3.132021314s) [noise=true , address=45.91.116.6:4992 , piece_id=6G7PY5YY32FTZBPRVLKSHSUH7KNYRIGHAFOPT7AVBCBM7HYH3MVQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.907199135 (13h11m3.132382192s) [noise=true , address=45.91.116.13:4169 , piece_id=7ND7C76WDTB77MRO3RPSQDGQFOK7H5WOBSD7RCVI7OUA5H4HSSXA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.910863127 (13h11m3.136046194s) [noise=true , address=104.36.82.23:4456 , piece_id=NOIRLI7AVEW4P3BRPJI27WOSFOO2WP2E6RDPFWWGHEWUKZ7HTF7Q, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.996930147 (13h11m3.222113204s) [upload=11582]: Next returned nil segment
Aug 26 05:14:47.996953811 (13h11m3.222136868s) [upload=11582]: Waiting for error group managing segments...
Aug 26 05:14:48.004359973 (13h11m3.229543019s) [noise=true , address=45.91.116.7:4030 , piece_id=JX6NWRZODJ3PEDOU43U5KQXHYBFYBPYJNDISY2T3DLK5NYZ5I7GA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.005508650 (13h11m3.230691697s) [noise=true , address=45.91.116.11:4146 , piece_id=HIWPDZR3CTF22YZCYH73QMJFSGET7QTBHRMNOLVMTSU6RJXSWTFA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.005522095 (13h11m3.230705142s) [noise=true , address=45.91.116.14:4203 , piece_id=C4YPVPOP56QR7P6RLTPZYXDBOJGNVSI7QS52DQRKESU6MBUEXN2A, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.006008057 (13h11m3.231191105s) [noise=true , address=45.91.116.13:4169 , piece_id=7ND7C76WDTB77MRO3RPSQDGQFOK7H5WOBSD7RCVI7OUA5H4HSSXA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.008257333 (13h11m3.23344037s) [noise=true , address=104.36.82.49:4987 , piece_id=ECBZZ23GJA5TWTPB2X7GZBFXIY5VL5FJR77KBF64QCO5BEVUALCQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.008662254 (13h11m3.233845311s) [noise=true , address=104.36.82.44:4873 , piece_id=CVFBS5TPDC7DSBY2MGZH4EL5DO3PLJDE2GJCZ2TL6JV7ETU2E34A, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.009403165 (13h11m3.234586212s) [noise=true , address=77.90.31.14:4694 , piece_id=O6K4VOHGODJGHAKXAGEKEAXU7ZXHIEVTE2S6KBKWSUJVZKACUWNQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.009804760 (13h11m3.234987806s) [noise=true , address=77.90.31.19:4855 , piece_id=V36EQSYCAE3O2GFK4KNVLKSYOJADKRIOMRBVSOHRI3RNL3AFEYNA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.011645477 (13h11m3.236828524s) [noise=true , address=104.36.82.13:4214 , piece_id=MBC7UIGS4Y7LUYQKQTG33CMKX7O25ANAV73ONFHCWKX7MLDIXOVA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.016428189 (13h11m3.241611427s) [noise=true , address=104.36.82.50:4030 , piece_id=FIQJULHAAWQQXVV3BTH5XIIYNUIVLOPXVEEOJWETTY6KBWKCX2WQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.016598520 (13h11m3.241781567s) [noise=true , address=104.36.82.10:4085 , piece_id=AZBDQYQUON4UVMJ3F4ULQXTNBB2OSWFFC72KU3MA2J7RGBJZELWA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.017570516 (13h11m3.242753553s) [noise=true , address=104.36.82.6:4891 , piece_id=HBARHJPGMLXHPEEA27KEMNMJVRFWTWR74YQL5XTO7KYRZ266VJXA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.030796789 (13h11m3.255979836s) [noise=true , address=104.36.82.60:4354 , piece_id=HFCYUWFHQ6AJA7REULM6KRFPMDFBBKRRJ22VQXSVTM5S4X3BX3PA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.089222399 (13h11m3.314405456s) [noise=true , address=104.36.82.23:4456 , piece_id=NOIRLI7AVEW4P3BRPJI27WOSFOO2WP2E6RDPFWWGHEWUKZ7HTF7Q, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.089247306 (13h11m3.314430354s) [seg_pos={0 0}, upload=11582]: Segment reached optimal threshold of 14 pieces.
Aug 26 05:14:48.089472019 (13h11m3.314655076s) [noise=true , address=45.91.116.6:4992 , piece_id=6G7PY5YY32FTZBPRVLKSHSUH7KNYRIGHAFOPT7AVBCBM7HYH3MVQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx): protocol: expected piece hash; context canceled; EOF
Aug 26 05:14:48.089527372 (13h11m3.314710429s) [noise=true , address=79.127.201.215:4349 , piece_id=BU6KUUSE77HTCME4DFHPPMADIFAFHI43PL2JPDEHZHZEYJMRXR4Q, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y1KYQJsvbNxfySU7mXySUmJtEDVAE7fst1XVRHU5N8Txmjn5T): protocol: write tcp 5.161.89.252:45582->79.127.201.215:4349: use of closed network connection; write tcp 5.161.89.252:45582->79.127.201.215:4349: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:45582->79.127.201.215:4349: use of closed network connection
Aug 26 05:14:48.089730384 (13h11m3.314913441s) [noise=true , address=79.127.205.246:4268 , piece_id=WMRJ2R3PWW4PEJKMI4WO6RDGV24VBMFDJMNA5ST4XXWNG3EZGYYA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k72iHWdcnM2JKs7VPTQFQwmpQ835HmAP8KpXxjMbEy5QxfzG): protocol: read tcp 5.161.89.252:10486->79.127.205.246:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 26 05:14:48.089765280 (13h11m3.314948327s) [noise=true , address=79.127.205.245:4244 , piece_id=G25ULKCGLHJCT7QCOP6ZSPXA7CHOS3KNIRZMRQT6NWRZPVRTCZCQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JFy3MxMFZYnjxJmujGpmcTdpVz6qLUMKf1aB3eYW236kCYgng): protocol: read tcp 5.161.89.252:25038->79.127.205.245:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 26 05:14:48.089773956 (13h11m3.314957003s) [seg_pos={0 0}, upload=11582]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 26 05:14:48.089836283 (13h11m3.31501934s) [upload=11582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 26 05:14:48.089842094 (13h11m3.315025141s) [upload=11582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 26 05:14:48.256048660 (13h11m3.481231707s) [upload=11582]: Done uploading segments...
Aug 26 05:14:48.256071674 (13h11m3.481254731s) [upload=11582]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 807.023781ms
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [.......................................X..........X....X....] duration:
178.781486ms remote:
"tcp:136.0.77.2:443" response headers:
X-Request-Id=VcCcTMkH9av Content-Length=423 Content-Type=application/xml Vary=Origin X-Amz-Request-Id=185F392FFCED1105 X-Xss-Protection=1; mode=block Date=Tue, 26 Aug 2025 05:14:18 GMT Accept-Ranges=bytes Content-Security-Policy=block-all-mixed-content Server=Storj target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-26 05:17:46 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis eval failed: dial tcp 10.37.0.83:6379: operation was canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>185F392FFCED1105</RequestId><HostId></HostId></Error>"
Duration: 178.800101ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [............................X..................XXX....XXX...] duration:
730.814909ms
FAIL (2025-08-26 05:17:44 UTC)
orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 429, RequestID: 185F393D8A8A6220, HostID: , api error SlowDown: Please reduce your request rate.
Duration: 730.834065ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................................................X.X..X.....] duration:
1.059690837s
FAIL (2025-08-26 05:17:39 UTC)
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 185F3927EAB52717, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled)
Duration: 1.059712468s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.........................XX.......X..X.......XXXXXXXX.XX....] duration:
3.320373154s
FAIL (2025-08-26 05:17:34 UTC)
operation: Commit uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 3.320425823s
core
uplink
us1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
5 out of 60 is failed (threshold=3) [................................................X.....XX.XX.] duration:
178.024504ms remote:
"tcp:45.38.24.2:443" response headers:
Accept-Ranges=bytes Content-Type=application/xml Server=Storj Vary=Origin X-Amz-Request-Id=185F394AB595C12F X-Request-Id=N1znMTMT8Va X-Xss-Protection=1; mode=block Content-Length=433 Content-Security-Policy=block-all-mixed-content Date=Tue, 26 Aug 2025 05:16:13 GMT target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-26 05:17:19 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis eval failed: dial tcp 10.37.0.83:6379: operation was canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks-us-select</BucketName><Resource>/hello-world.txt</Resource><RequestId>185F394AB595C12F</RequestId><HostId></HostId></Error>"
Duration: 178.039511ms
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..........................XX.......X..X.......XXXXXXXX.XX...] duration:
2.123073578s
FAIL (2025-08-26 05:17:16 UTC)
operation: Commit uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 2.123100109s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [................................X....XX.........XX.XX..XXX..] duration:
741.219562ms
FAIL (2025-08-26 05:17:16 UTC)
orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 185F3945025C03C0, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled)
Duration: 741.233708ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
9 out of 60 is failed (threshold=3) [...........................X.....XX..........XX....X..X.XX..] duration:
654.509635ms
FAIL (2025-08-26 05:17:15 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 654.563817ms
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [............................X.....X...........X.X.XX.XXXXX..] duration:
3.088664069s
FAIL (2025-08-26 05:17:10 UTC)
operation: DownloadObject
======ERROR======
uplink: metaclient: accounting backend: Redis eval failed: dial tcp 10.37.0.83:6379: operation was canceled
======LOGS======
Aug 26 05:15:11.856866547 (13h11m27.082049594s) [upload=11590]: Starting upload
Aug 26 05:15:11.856947388 (13h11m27.082130445s) [upload=11590]: Done starting upload
Aug 26 05:15:11.857007762 (13h11m27.082190808s) [upload=11590]: Uploading segments...
Aug 26 05:15:11.857250268 (13h11m27.082433315s) [upload=11590]: Got next segment. Inline: false
Aug 26 05:15:11.857286635 (13h11m27.082469692s) [upload=11590]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 26 05:15:11.857293298 (13h11m27.082476355s) [upload=11590]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 26 05:15:22.858314105 (13h11m38.083497183s) [seg_pos={0 0}, upload=11590]: Begin upload segment...
Aug 26 05:15:22.858367536 (13h11m38.083550623s) [seg_pos={0 0}, upload=11590]: Done begin upload segment.
Aug 26 05:15:22.858400479 (13h11m38.083583556s) [noise=true , address=45.91.116.14:4190 , piece_id=4R7WJUCTCWBXGOWT26AKRAJJJGVAWVOQWQDI5CUAXWAGKBO3LUSA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.858456674 (13h11m38.083639761s) [noise=true , address=77.90.31.15:4699 , piece_id=DK3JB4BSU5D4DGRDKIGEYH4J2XFGGHQGSYLDLG76OIILI5QQF44Q, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.858923801 (13h11m38.084106879s) [noise=true , address=45.91.116.5:4298 , piece_id=XABSOODUDZQAMVECL6IVUVWMQFZX3IAOZU7D4UHF3KIHP3B3CMPA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.859079042 (13h11m38.08426213s) [noise=true , address=45.91.116.9:4261 , piece_id=PK26MWMBLSKNWB2PMTGGAF6NZ4EAY5BIUQP3CVGYOXRXI2NEWK4Q, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.859314645 (13h11m38.084497722s) [noise=true , address=45.91.116.13:4155 , piece_id=2ZPF5WCMSCW3S2I2RC2EVJIDNFKH775TIE3J7Q3AJRHR5JJKBJ3A, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.859514331 (13h11m38.084697417s) [noise=true , address=45.91.116.13:4154 , piece_id=UVDYGWF7HRYICI4YDHO2GEBAG5YJX24MFKIVGIVJMFAGKH6IQIOQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.859739903 (13h11m38.084922981s) [noise=true , address=45.91.116.11:4118 , piece_id=XADWBNSBO7PIOWZROEFOVQVYZBY7DXARHZOYDIOLBA54ZPFU4NJQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860027705 (13h11m38.085210792s) [noise=true , address=104.36.82.7:4909 , piece_id=7YOQSRLJITDS42AURAM4BP24GSE42OPR4JCNLQ6GD6Y6M5ZZLMIQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860244512 (13h11m38.085427589s) [noise=true , address=77.90.31.13:4626 , piece_id=6OVHSO55T4UTUWVLDB4WT2QLK5U3JTIDI3ACVPYZB4LUHTBOZ3EQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860413158 (13h11m38.085596236s) [noise=true , address=45.91.116.9:4263 , piece_id=YPCUKYKUKEDWDZFRYMNCULCMUAOVLYA3DTZJU7KRFUGPQCE5Z6ZA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860662036 (13h11m38.085845133s) [noise=true , address=104.36.82.12:4150 , piece_id=Y4AJKPQYW5A2PSV5F6IZ27C6XE2BYU7F7OUWGNAQXDKCZPOTGDHA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.860962371 (13h11m38.086145457s) [noise=true , address=77.90.31.21:4879 , piece_id=4I3TJN7NUHC2QOJ6JLLSXTM6SQ7UMEQCWYBIRNYNRAJYKENJBQQQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.861160522 (13h11m38.0863436s) [noise=true , address=104.36.82.2:4729 , piece_id=YBU6CLRMUTMBQJMBTQ7J4YBXPDGDSAIKTY36HWSGBXOXTY6UPKGA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.861340972 (13h11m38.086524019s) [noise=true , address=77.90.31.14:4663 , piece_id=RM3L6KVSXK7XPWIS2PB63PBSKC4XF22EKHY3PAJE264NNG65TRHQ, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.861443835 (13h11m38.086626881s) [noise=true , address=45.91.116.4:4939 , piece_id=FKDXX5QA7VPHPQ5WW4W3OJCKBFISFTGZHQSOPCF2LKUZSFOJHLHA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.861699475 (13h11m38.086882522s) [noise=true , address=77.90.31.11:4554 , piece_id=DXNENQZMA57JJ4QB46OSDDVN6RRTTT6DAWNJ3JTUWWHYL2XUA7GA, seg_pos={0 0}, upload=11590]: Uploading piece...
Aug 26 05:15:22.990256495 (13h11m38.215439542s) [upload=11590]: Next returned nil segment
Aug 26 05:15:22.990276573 (13h11m38.21545963s) [upload=11590]: Waiting for error group managing segments...
Aug 26 05:15:22.998364304 (13h11m38.223547351s) [noise=true , address=45.91.116.9:4261 , piece_id=PK26MWMBLSKNWB2PMTGGAF6NZ4EAY5BIUQP3CVGYOXRXI2NEWK4Q, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.998564650 (13h11m38.223747697s) [noise=true , address=45.91.116.14:4190 , piece_id=4R7WJUCTCWBXGOWT26AKRAJJJGVAWVOQWQDI5CUAXWAGKBO3LUSA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.998988045 (13h11m38.224171092s) [noise=true , address=45.91.116.11:4118 , piece_id=XADWBNSBO7PIOWZROEFOVQVYZBY7DXARHZOYDIOLBA54ZPFU4NJQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.999143488 (13h11m38.224326535s) [noise=true , address=45.91.116.9:4263 , piece_id=YPCUKYKUKEDWDZFRYMNCULCMUAOVLYA3DTZJU7KRFUGPQCE5Z6ZA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.999571281 (13h11m38.224754328s) [noise=true , address=77.90.31.13:4626 , piece_id=6OVHSO55T4UTUWVLDB4WT2QLK5U3JTIDI3ACVPYZB4LUHTBOZ3EQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:22.999598783 (13h11m38.22478183s) [noise=true , address=45.91.116.13:4154 , piece_id=UVDYGWF7HRYICI4YDHO2GEBAG5YJX24MFKIVGIVJMFAGKH6IQIOQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.000104903 (13h11m38.22528795s) [noise=true , address=45.91.116.5:4298 , piece_id=XABSOODUDZQAMVECL6IVUVWMQFZX3IAOZU7D4UHF3KIHP3B3CMPA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.000133807 (13h11m38.225316864s) [noise=true , address=77.90.31.21:4879 , piece_id=4I3TJN7NUHC2QOJ6JLLSXTM6SQ7UMEQCWYBIRNYNRAJYKENJBQQQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.000584494 (13h11m38.225767551s) [noise=true , address=45.91.116.13:4155 , piece_id=2ZPF5WCMSCW3S2I2RC2EVJIDNFKH775TIE3J7Q3AJRHR5JJKBJ3A, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.000818082 (13h11m38.226001139s) [noise=true , address=45.91.116.4:4939 , piece_id=FKDXX5QA7VPHPQ5WW4W3OJCKBFISFTGZHQSOPCF2LKUZSFOJHLHA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.001007669 (13h11m38.226190736s) [noise=true , address=104.36.82.7:4909 , piece_id=7YOQSRLJITDS42AURAM4BP24GSE42OPR4JCNLQ6GD6Y6M5ZZLMIQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.001806589 (13h11m38.226989646s) [noise=true , address=77.90.31.14:4663 , piece_id=RM3L6KVSXK7XPWIS2PB63PBSKC4XF22EKHY3PAJE264NNG65TRHQ, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.001962061 (13h11m38.227145108s) [noise=true , address=77.90.31.15:4699 , piece_id=DK3JB4BSU5D4DGRDKIGEYH4J2XFGGHQGSYLDLG76OIILI5QQF44Q, seg_pos={0 0}, upload=11590]: Done uploading piece. err: <nil>
Aug 26 05:15:23.001984242 (13h11m38.22716732s) [seg_pos={0 0}, upload=11590]: Segment reached optimal threshold of 13 pieces.
Aug 26 05:15:23.002128384 (13h11m38.227311431s) [noise=true , address=104.36.82.12:4150 , piece_id=Y4AJKPQYW5A2PSV5F6IZ27C6XE2BYU7F7OUWGNAQXDKCZPOTGDHA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF
Aug 26 05:15:23.002141278 (13h11m38.227324335s) [noise=true , address=77.90.31.11:4554 , piece_id=DXNENQZMA57JJ4QB46OSDDVN6RRTTT6DAWNJ3JTUWWHYL2XUA7GA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF
Aug 26 05:15:23.002152920 (13h11m38.227335967s) [noise=true , address=104.36.82.2:4729 , piece_id=YBU6CLRMUTMBQJMBTQ7J4YBXPDGDSAIKTY36HWSGBXOXTY6UPKGA, seg_pos={0 0}, upload=11590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF
Aug 26 05:15:23.002175222 (13h11m38.227358269s) [seg_pos={0 0}, upload=11590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 26 05:15:23.002254370 (13h11m38.227437437s) [upload=11590]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 26 05:15:23.002260071 (13h11m38.227443118s) [upload=11590]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 26 05:15:31.060086659 (13h11m46.285269727s) [upload=11590]: Done uploading segments...
Aug 26 05:15:31.060104984 (13h11m46.285288061s) [upload=11590]: Upload finished. err: <nil>
Duration: 3.088792609s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
4 out of 60 is failed (threshold=6) [..................................X...............X..X....X.] duration:
242.14968ms remote:
"tcp:136.0.77.2:443" response headers:
Access-Control-Allow-Methods=GET, HEAD Access-Control-Allow-Origin=* Date=Tue, 26 Aug 2025 05:15:54 GMT Accept-Ranges=bytes X-Request-Id=PzQG5qDgV69 Content-Type=text/html; charset=utf-8 Access-Control-Allow-Headers=* target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt
FAIL (2025-08-26 05:17:03 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<!DOCTYPE html>\n<html lang=\"en\">\n\n<head>\n <meta charset=\"utf-8\">\n <title>Error | Storj</title>\n\n <meta property=\"og:url\" content=\"https://link.storjshare.io\" />\n <meta property=\"og:title\" content=\"Error | Storj\" />\n <meta property=\"og:description\" name=\"description\" content=\"Shared content - Storj\" />\n \n <meta property=\"og:image\" content=\"https://link.storjshare.io/static/img/file.png?v=f0d1e95\" />\n \n\n <meta name=\"twitter:card\" content=\"summary\" />\n <meta name=\"twitter:site\" content=\"@storj\" />\n <meta name=\"twitter:title\" content=\"Error | Storj\" />\n <meta name=\"twitter:description\" content=\"Shared content - Storj\" />\n \n <meta name=\"twitter:image\" content=\"https://link.storjshare.io/static/img/file.png?v=f0d1e95\" />\n \n\n <link rel=\"shortcut icon\"\n href=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAIAAAD8GO2jAAAC40lEQVRIDe1US0iUURQ+V8fJmcnykTJmNA0tMszETUVlCrkoR+2lWdFDk7AHlNQiCGaREhlUli3ETSRFqx4UlAi5ySgqS8hQiXLMTKca35k6j3O6d5zE//rnuNCF4Mdd3HvO+c53/3P/cxgRwUwiY"
Duration: 242.167744ms
edge
linksharing
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
4 out of 60 is failed (threshold=3) [..................................................X.X..XX...] duration:
2.224675494s
FAIL (2025-08-26 05:16:58 UTC)
operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled ======LOGS====== Aug 26 05:13:43.987058282 (13h9m59.212241339s) [upload=11567]: Starting upload Aug 26 05:13:43.987144814 (13h9m59.212327871s) [upload=11567]: Done starting upload Aug 26 05:13:43.987212292 (13h9m59.212395349s) [upload=11567]: Uploading segments... Aug 26 05:13:43.987257306 (13h9m59.212440363s) [upload=11567]: Got next segment. Inline: true Aug 26 05:13:43.987263478 (13h9m59.212446525s) [upload=11567]: Waiting for error group managing segments... Aug 26 05:13:43.987304054 (13h9m59.212487152s) [upload=11567]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 26 05:13:43.987309574 (13h9m59.212492631s) [upload=11567]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 26 05:13:43.987313713 (13h9m59.21249677s) [upload=11567]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 26 05:14:09.912697999 (13h10m25.137881087s) [upload=11567]: Done uploading segments... Aug 26 05:14:09.912726052 (13h10m25.137909209s) [upload=11567]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 2.224696995s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
9 out of 60 is failed (threshold=3) [.............................XX.......X..........XX.X.X.XX..] duration:
515.321096ms
FAIL (2025-08-26 05:16:57 UTC)
operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled ======LOGS====== Aug 26 05:14:29.029036501 (13h10m44.254219628s) [upload=11579]: Starting upload Aug 26 05:14:29.029135167 (13h10m44.254318244s) [upload=11579]: Done starting upload Aug 26 05:14:29.029207823 (13h10m44.25439091s) [upload=11579]: Uploading segments... Aug 26 05:14:29.029253299 (13h10m44.254436376s) [upload=11579]: Got next segment. Inline: true Aug 26 05:14:29.029259290 (13h10m44.254442367s) [upload=11579]: Waiting for error group managing segments... Aug 26 05:14:29.029307520 (13h10m44.254490598s) [upload=11579]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 26 05:14:29.029313171 (13h10m44.254496248s) [upload=11579]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 26 05:14:29.029317299 (13h10m44.254500376s) [upload=11579]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 26 05:14:41.554926683 (13h10m56.78010974s) [upload=11579]: Done uploading segments... Aug 26 05:14:41.554940789 (13h10m56.780123846s) [upload=11579]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 515.339982ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
9 out of 60 is failed (threshold=3) [............................X.....XX..........XX....X..X.XX.] duration:
602.951463ms
FAIL (2025-08-26 05:16:55 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 602.977853ms
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [........................................X........XX.X.X.X...] duration:
6.515579201s
FAIL (2025-08-26 05:16:52 UTC)
operation: Commit
======ERROR======
uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
======LOGS======
0704832 (13h11m2.905887889s) [upload=11582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 26 05:14:47.902776638 (13h11m3.127959686s) [seg_pos={0 0}, upload=11582]: Begin upload segment...
Aug 26 05:14:47.902820661 (13h11m3.128003709s) [seg_pos={0 0}, upload=11582]: Done begin upload segment.
Aug 26 05:14:47.902849054 (13h11m3.128032111s) [noise=true , address=45.91.116.11:4146 , piece_id=HIWPDZR3CTF22YZCYH73QMJFSGET7QTBHRMNOLVMTSU6RJXSWTFA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.902873490 (13h11m3.128056537s) [noise=true , address=77.90.31.14:4694 , piece_id=O6K4VOHGODJGHAKXAGEKEAXU7ZXHIEVTE2S6KBKWSUJVZKACUWNQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903189514 (13h11m3.128372571s) [noise=true , address=104.36.82.13:4214 , piece_id=MBC7UIGS4Y7LUYQKQTG33CMKX7O25ANAV73ONFHCWKX7MLDIXOVA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903342101 (13h11m3.128525148s) [noise=true , address=45.91.116.14:4203 , piece_id=C4YPVPOP56QR7P6RLTPZYXDBOJGNVSI7QS52DQRKESU6MBUEXN2A, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903413926 (13h11m3.128596973s) [noise=true , address=104.36.82.50:4030 , piece_id=FIQJULHAAWQQXVV3BTH5XIIYNUIVLOPXVEEOJWETTY6KBWKCX2WQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903641763 (13h11m3.12882482s) [noise=true , address=79.127.205.246:4268 , piece_id=WMRJ2R3PWW4PEJKMI4WO6RDGV24VBMFDJMNA5ST4XXWNG3EZGYYA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903697358 (13h11m3.128880406s) [noise=true , address=104.36.82.44:4873 , piece_id=CVFBS5TPDC7DSBY2MGZH4EL5DO3PLJDE2GJCZ2TL6JV7ETU2E34A, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.904382555 (13h11m3.129565612s) [noise=true , address=77.90.31.19:4855 , piece_id=V36EQSYCAE3O2GFK4KNVLKSYOJADKRIOMRBVSOHRI3RNL3AFEYNA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.904391010 (13h11m3.129574068s) [noise=true , address=104.36.82.6:4891 , piece_id=HBARHJPGMLXHPEEA27KEMNMJVRFWTWR74YQL5XTO7KYRZ266VJXA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.904787035 (13h11m3.129970082s) [noise=true , address=104.36.82.60:4354 , piece_id=HFCYUWFHQ6AJA7REULM6KRFPMDFBBKRRJ22VQXSVTM5S4X3BX3PA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.905089312 (13h11m3.130272359s) [noise=true , address=104.36.82.49:4987 , piece_id=ECBZZ23GJA5TWTPB2X7GZBFXIY5VL5FJR77KBF64QCO5BEVUALCQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.905389606 (13h11m3.130572683s) [noise=true , address=104.36.82.10:4085 , piece_id=AZBDQYQUON4UVMJ3F4ULQXTNBB2OSWFFC72KU3MA2J7RGBJZELWA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.905880078 (13h11m3.131063125s) [noise=true , address=79.127.201.215:4349 , piece_id=BU6KUUSE77HTCME4DFHPPMADIFAFHI43PL2JPDEHZHZEYJMRXR4Q, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.906031132 (13h11m3.131214179s) [noise=true , address=45.91.116.7:4030 , piece_id=JX6NWRZODJ3PEDOU43U5KQXHYBFYBPYJNDISY2T3DLK5NYZ5I7GA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.906575734 (13h11m3.131758791s) [noise=true , address=79.127.205.245:4244 , piece_id=G25ULKCGLHJCT7QCOP6ZSPXA7CHOS3KNIRZMRQT6NWRZPVRTCZCQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.906838247 (13h11m3.132021314s) [noise=true , address=45.91.116.6:4992 , piece_id=6G7PY5YY32FTZBPRVLKSHSUH7KNYRIGHAFOPT7AVBCBM7HYH3MVQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.907199135 (13h11m3.132382192s) [noise=true , address=45.91.116.13:4169 , piece_id=7ND7C76WDTB77MRO3RPSQDGQFOK7H5WOBSD7RCVI7OUA5H4HSSXA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.910863127 (13h11m3.136046194s) [noise=true , address=104.36.82.23:4456 , piece_id=NOIRLI7AVEW4P3BRPJI27WOSFOO2WP2E6RDPFWWGHEWUKZ7HTF7Q, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.996930147 (13h11m3.222113204s) [upload=11582]: Next returned nil segment
Aug 26 05:14:47.996953811 (13h11m3.222136868s) [upload=11582]: Waiting for error group managing segments...
Aug 26 05:14:48.004359973 (13h11m3.229543019s) [noise=true , address=45.91.116.7:4030 , piece_id=JX6NWRZODJ3PEDOU43U5KQXHYBFYBPYJNDISY2T3DLK5NYZ5I7GA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.005508650 (13h11m3.230691697s) [noise=true , address=45.91.116.11:4146 , piece_id=HIWPDZR3CTF22YZCYH73QMJFSGET7QTBHRMNOLVMTSU6RJXSWTFA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.005522095 (13h11m3.230705142s) [noise=true , address=45.91.116.14:4203 , piece_id=C4YPVPOP56QR7P6RLTPZYXDBOJGNVSI7QS52DQRKESU6MBUEXN2A, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.006008057 (13h11m3.231191105s) [noise=true , address=45.91.116.13:4169 , piece_id=7ND7C76WDTB77MRO3RPSQDGQFOK7H5WOBSD7RCVI7OUA5H4HSSXA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.008257333 (13h11m3.23344037s) [noise=true , address=104.36.82.49:4987 , piece_id=ECBZZ23GJA5TWTPB2X7GZBFXIY5VL5FJR77KBF64QCO5BEVUALCQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.008662254 (13h11m3.233845311s) [noise=true , address=104.36.82.44:4873 , piece_id=CVFBS5TPDC7DSBY2MGZH4EL5DO3PLJDE2GJCZ2TL6JV7ETU2E34A, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.009403165 (13h11m3.234586212s) [noise=true , address=77.90.31.14:4694 , piece_id=O6K4VOHGODJGHAKXAGEKEAXU7ZXHIEVTE2S6KBKWSUJVZKACUWNQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.009804760 (13h11m3.234987806s) [noise=true , address=77.90.31.19:4855 , piece_id=V36EQSYCAE3O2GFK4KNVLKSYOJADKRIOMRBVSOHRI3RNL3AFEYNA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.011645477 (13h11m3.236828524s) [noise=true , address=104.36.82.13:4214 , piece_id=MBC7UIGS4Y7LUYQKQTG33CMKX7O25ANAV73ONFHCWKX7MLDIXOVA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.016428189 (13h11m3.241611427s) [noise=true , address=104.36.82.50:4030 , piece_id=FIQJULHAAWQQXVV3BTH5XIIYNUIVLOPXVEEOJWETTY6KBWKCX2WQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.016598520 (13h11m3.241781567s) [noise=true , address=104.36.82.10:4085 , piece_id=AZBDQYQUON4UVMJ3F4ULQXTNBB2OSWFFC72KU3MA2J7RGBJZELWA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.017570516 (13h11m3.242753553s) [noise=true , address=104.36.82.6:4891 , piece_id=HBARHJPGMLXHPEEA27KEMNMJVRFWTWR74YQL5XTO7KYRZ266VJXA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.030796789 (13h11m3.255979836s) [noise=true , address=104.36.82.60:4354 , piece_id=HFCYUWFHQ6AJA7REULM6KRFPMDFBBKRRJ22VQXSVTM5S4X3BX3PA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.089222399 (13h11m3.314405456s) [noise=true , address=104.36.82.23:4456 , piece_id=NOIRLI7AVEW4P3BRPJI27WOSFOO2WP2E6RDPFWWGHEWUKZ7HTF7Q, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.089247306 (13h11m3.314430354s) [seg_pos={0 0}, upload=11582]: Segment reached optimal threshold of 14 pieces.
Aug 26 05:14:48.089472019 (13h11m3.314655076s) [noise=true , address=45.91.116.6:4992 , piece_id=6G7PY5YY32FTZBPRVLKSHSUH7KNYRIGHAFOPT7AVBCBM7HYH3MVQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx): protocol: expected piece hash; context canceled; EOF
Aug 26 05:14:48.089527372 (13h11m3.314710429s) [noise=true , address=79.127.201.215:4349 , piece_id=BU6KUUSE77HTCME4DFHPPMADIFAFHI43PL2JPDEHZHZEYJMRXR4Q, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y1KYQJsvbNxfySU7mXySUmJtEDVAE7fst1XVRHU5N8Txmjn5T): protocol: write tcp 5.161.89.252:45582->79.127.201.215:4349: use of closed network connection; write tcp 5.161.89.252:45582->79.127.201.215:4349: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:45582->79.127.201.215:4349: use of closed network connection
Aug 26 05:14:48.089730384 (13h11m3.314913441s) [noise=true , address=79.127.205.246:4268 , piece_id=WMRJ2R3PWW4PEJKMI4WO6RDGV24VBMFDJMNA5ST4XXWNG3EZGYYA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k72iHWdcnM2JKs7VPTQFQwmpQ835HmAP8KpXxjMbEy5QxfzG): protocol: read tcp 5.161.89.252:10486->79.127.205.246:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 26 05:14:48.089765280 (13h11m3.314948327s) [noise=true , address=79.127.205.245:4244 , piece_id=G25ULKCGLHJCT7QCOP6ZSPXA7CHOS3KNIRZMRQT6NWRZPVRTCZCQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JFy3MxMFZYnjxJmujGpmcTdpVz6qLUMKf1aB3eYW236kCYgng): protocol: read tcp 5.161.89.252:25038->79.127.205.245:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 26 05:14:48.089773956 (13h11m3.314957003s) [seg_pos={0 0}, upload=11582]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 26 05:14:48.089836283 (13h11m3.31501934s) [upload=11582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 26 05:14:48.089842094 (13h11m3.315025141s) [upload=11582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 26 05:14:48.256048660 (13h11m3.481231707s) [upload=11582]: Done uploading segments...
Aug 26 05:14:48.256071674 (13h11m3.481254731s) [upload=11582]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 6.51562673s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.............................X..................XXX....XXX..] duration:
15.595242386s
FAIL (2025-08-26 05:16:51 UTC)
orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 429, RequestID: 185F393D8A8A6220, HostID: , api error SlowDown: Please reduce your request rate.
Duration: 15.595251053s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................................................X.X..X....] duration:
19.232292155s
FAIL (2025-08-26 05:16:51 UTC)
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 185F3927EAB52717, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled)
Duration: 19.232578113s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [........................................X..........X....X...] duration:
8.040319442s remote:
"tcp:136.0.77.2:443" response headers:
X-Request-Id=VcCcTMkH9av Content-Length=423 Content-Type=application/xml Vary=Origin X-Amz-Request-Id=185F392FFCED1105 X-Xss-Protection=1; mode=block Date=Tue, 26 Aug 2025 05:14:18 GMT Accept-Ranges=bytes Content-Security-Policy=block-all-mixed-content Server=Storj target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-26 05:16:42 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis eval failed: dial tcp 10.37.0.83:6379: operation was canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>185F392FFCED1105</RequestId><HostId></HostId></Error>"
Duration: 8.040347956s
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [.................................X....XX.........XX.XX..XXX.] duration:
19.955783751s
FAIL (2025-08-26 05:16:39 UTC)
orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 185F3945025C03C0, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled)
Duration: 19.955808968s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
14 out of 60 is failed (threshold=3) [...........................XX.......X..X.......XXXXXXXX.XX..] duration:
17.184727754s
FAIL (2025-08-26 05:16:35 UTC)
operation: Commit uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 17.18474154s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
9 out of 60 is failed (threshold=3) [.............................X.....XX..........XX....X..X.XX] duration:
30.000669438s
FAIL (2025-08-26 05:16:27 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000715376s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........................................X........XX.X.X.X..] duration:
9.780272149s
FAIL (2025-08-26 05:16:23 UTC)
operation: Commit
======ERROR======
uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
======LOGS======
0704832 (13h11m2.905887889s) [upload=11582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 26 05:14:47.902776638 (13h11m3.127959686s) [seg_pos={0 0}, upload=11582]: Begin upload segment...
Aug 26 05:14:47.902820661 (13h11m3.128003709s) [seg_pos={0 0}, upload=11582]: Done begin upload segment.
Aug 26 05:14:47.902849054 (13h11m3.128032111s) [noise=true , address=45.91.116.11:4146 , piece_id=HIWPDZR3CTF22YZCYH73QMJFSGET7QTBHRMNOLVMTSU6RJXSWTFA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.902873490 (13h11m3.128056537s) [noise=true , address=77.90.31.14:4694 , piece_id=O6K4VOHGODJGHAKXAGEKEAXU7ZXHIEVTE2S6KBKWSUJVZKACUWNQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903189514 (13h11m3.128372571s) [noise=true , address=104.36.82.13:4214 , piece_id=MBC7UIGS4Y7LUYQKQTG33CMKX7O25ANAV73ONFHCWKX7MLDIXOVA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903342101 (13h11m3.128525148s) [noise=true , address=45.91.116.14:4203 , piece_id=C4YPVPOP56QR7P6RLTPZYXDBOJGNVSI7QS52DQRKESU6MBUEXN2A, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903413926 (13h11m3.128596973s) [noise=true , address=104.36.82.50:4030 , piece_id=FIQJULHAAWQQXVV3BTH5XIIYNUIVLOPXVEEOJWETTY6KBWKCX2WQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903641763 (13h11m3.12882482s) [noise=true , address=79.127.205.246:4268 , piece_id=WMRJ2R3PWW4PEJKMI4WO6RDGV24VBMFDJMNA5ST4XXWNG3EZGYYA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.903697358 (13h11m3.128880406s) [noise=true , address=104.36.82.44:4873 , piece_id=CVFBS5TPDC7DSBY2MGZH4EL5DO3PLJDE2GJCZ2TL6JV7ETU2E34A, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.904382555 (13h11m3.129565612s) [noise=true , address=77.90.31.19:4855 , piece_id=V36EQSYCAE3O2GFK4KNVLKSYOJADKRIOMRBVSOHRI3RNL3AFEYNA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.904391010 (13h11m3.129574068s) [noise=true , address=104.36.82.6:4891 , piece_id=HBARHJPGMLXHPEEA27KEMNMJVRFWTWR74YQL5XTO7KYRZ266VJXA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.904787035 (13h11m3.129970082s) [noise=true , address=104.36.82.60:4354 , piece_id=HFCYUWFHQ6AJA7REULM6KRFPMDFBBKRRJ22VQXSVTM5S4X3BX3PA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.905089312 (13h11m3.130272359s) [noise=true , address=104.36.82.49:4987 , piece_id=ECBZZ23GJA5TWTPB2X7GZBFXIY5VL5FJR77KBF64QCO5BEVUALCQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.905389606 (13h11m3.130572683s) [noise=true , address=104.36.82.10:4085 , piece_id=AZBDQYQUON4UVMJ3F4ULQXTNBB2OSWFFC72KU3MA2J7RGBJZELWA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.905880078 (13h11m3.131063125s) [noise=true , address=79.127.201.215:4349 , piece_id=BU6KUUSE77HTCME4DFHPPMADIFAFHI43PL2JPDEHZHZEYJMRXR4Q, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.906031132 (13h11m3.131214179s) [noise=true , address=45.91.116.7:4030 , piece_id=JX6NWRZODJ3PEDOU43U5KQXHYBFYBPYJNDISY2T3DLK5NYZ5I7GA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.906575734 (13h11m3.131758791s) [noise=true , address=79.127.205.245:4244 , piece_id=G25ULKCGLHJCT7QCOP6ZSPXA7CHOS3KNIRZMRQT6NWRZPVRTCZCQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.906838247 (13h11m3.132021314s) [noise=true , address=45.91.116.6:4992 , piece_id=6G7PY5YY32FTZBPRVLKSHSUH7KNYRIGHAFOPT7AVBCBM7HYH3MVQ, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.907199135 (13h11m3.132382192s) [noise=true , address=45.91.116.13:4169 , piece_id=7ND7C76WDTB77MRO3RPSQDGQFOK7H5WOBSD7RCVI7OUA5H4HSSXA, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.910863127 (13h11m3.136046194s) [noise=true , address=104.36.82.23:4456 , piece_id=NOIRLI7AVEW4P3BRPJI27WOSFOO2WP2E6RDPFWWGHEWUKZ7HTF7Q, seg_pos={0 0}, upload=11582]: Uploading piece...
Aug 26 05:14:47.996930147 (13h11m3.222113204s) [upload=11582]: Next returned nil segment
Aug 26 05:14:47.996953811 (13h11m3.222136868s) [upload=11582]: Waiting for error group managing segments...
Aug 26 05:14:48.004359973 (13h11m3.229543019s) [noise=true , address=45.91.116.7:4030 , piece_id=JX6NWRZODJ3PEDOU43U5KQXHYBFYBPYJNDISY2T3DLK5NYZ5I7GA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.005508650 (13h11m3.230691697s) [noise=true , address=45.91.116.11:4146 , piece_id=HIWPDZR3CTF22YZCYH73QMJFSGET7QTBHRMNOLVMTSU6RJXSWTFA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.005522095 (13h11m3.230705142s) [noise=true , address=45.91.116.14:4203 , piece_id=C4YPVPOP56QR7P6RLTPZYXDBOJGNVSI7QS52DQRKESU6MBUEXN2A, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.006008057 (13h11m3.231191105s) [noise=true , address=45.91.116.13:4169 , piece_id=7ND7C76WDTB77MRO3RPSQDGQFOK7H5WOBSD7RCVI7OUA5H4HSSXA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.008257333 (13h11m3.23344037s) [noise=true , address=104.36.82.49:4987 , piece_id=ECBZZ23GJA5TWTPB2X7GZBFXIY5VL5FJR77KBF64QCO5BEVUALCQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.008662254 (13h11m3.233845311s) [noise=true , address=104.36.82.44:4873 , piece_id=CVFBS5TPDC7DSBY2MGZH4EL5DO3PLJDE2GJCZ2TL6JV7ETU2E34A, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.009403165 (13h11m3.234586212s) [noise=true , address=77.90.31.14:4694 , piece_id=O6K4VOHGODJGHAKXAGEKEAXU7ZXHIEVTE2S6KBKWSUJVZKACUWNQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.009804760 (13h11m3.234987806s) [noise=true , address=77.90.31.19:4855 , piece_id=V36EQSYCAE3O2GFK4KNVLKSYOJADKRIOMRBVSOHRI3RNL3AFEYNA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.011645477 (13h11m3.236828524s) [noise=true , address=104.36.82.13:4214 , piece_id=MBC7UIGS4Y7LUYQKQTG33CMKX7O25ANAV73ONFHCWKX7MLDIXOVA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.016428189 (13h11m3.241611427s) [noise=true , address=104.36.82.50:4030 , piece_id=FIQJULHAAWQQXVV3BTH5XIIYNUIVLOPXVEEOJWETTY6KBWKCX2WQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.016598520 (13h11m3.241781567s) [noise=true , address=104.36.82.10:4085 , piece_id=AZBDQYQUON4UVMJ3F4ULQXTNBB2OSWFFC72KU3MA2J7RGBJZELWA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.017570516 (13h11m3.242753553s) [noise=true , address=104.36.82.6:4891 , piece_id=HBARHJPGMLXHPEEA27KEMNMJVRFWTWR74YQL5XTO7KYRZ266VJXA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.030796789 (13h11m3.255979836s) [noise=true , address=104.36.82.60:4354 , piece_id=HFCYUWFHQ6AJA7REULM6KRFPMDFBBKRRJ22VQXSVTM5S4X3BX3PA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.089222399 (13h11m3.314405456s) [noise=true , address=104.36.82.23:4456 , piece_id=NOIRLI7AVEW4P3BRPJI27WOSFOO2WP2E6RDPFWWGHEWUKZ7HTF7Q, seg_pos={0 0}, upload=11582]: Done uploading piece. err: <nil>
Aug 26 05:14:48.089247306 (13h11m3.314430354s) [seg_pos={0 0}, upload=11582]: Segment reached optimal threshold of 14 pieces.
Aug 26 05:14:48.089472019 (13h11m3.314655076s) [noise=true , address=45.91.116.6:4992 , piece_id=6G7PY5YY32FTZBPRVLKSHSUH7KNYRIGHAFOPT7AVBCBM7HYH3MVQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx): protocol: expected piece hash; context canceled; EOF
Aug 26 05:14:48.089527372 (13h11m3.314710429s) [noise=true , address=79.127.201.215:4349 , piece_id=BU6KUUSE77HTCME4DFHPPMADIFAFHI43PL2JPDEHZHZEYJMRXR4Q, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y1KYQJsvbNxfySU7mXySUmJtEDVAE7fst1XVRHU5N8Txmjn5T): protocol: write tcp 5.161.89.252:45582->79.127.201.215:4349: use of closed network connection; write tcp 5.161.89.252:45582->79.127.201.215:4349: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:45582->79.127.201.215:4349: use of closed network connection
Aug 26 05:14:48.089730384 (13h11m3.314913441s) [noise=true , address=79.127.205.246:4268 , piece_id=WMRJ2R3PWW4PEJKMI4WO6RDGV24VBMFDJMNA5ST4XXWNG3EZGYYA, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k72iHWdcnM2JKs7VPTQFQwmpQ835HmAP8KpXxjMbEy5QxfzG): protocol: read tcp 5.161.89.252:10486->79.127.205.246:4268: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 26 05:14:48.089765280 (13h11m3.314948327s) [noise=true , address=79.127.205.245:4244 , piece_id=G25ULKCGLHJCT7QCOP6ZSPXA7CHOS3KNIRZMRQT6NWRZPVRTCZCQ, seg_pos={0 0}, upload=11582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JFy3MxMFZYnjxJmujGpmcTdpVz6qLUMKf1aB3eYW236kCYgng): protocol: read tcp 5.161.89.252:25038->79.127.205.245:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 26 05:14:48.089773956 (13h11m3.314957003s) [seg_pos={0 0}, upload=11582]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 26 05:14:48.089836283 (13h11m3.31501934s) [upload=11582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 26 05:14:48.089842094 (13h11m3.315025141s) [upload=11582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 26 05:14:48.256048660 (13h11m3.481231707s) [upload=11582]: Done uploading segments...
Aug 26 05:14:48.256071674 (13h11m3.481254731s) [upload=11582]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 9.780292667s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
14 out of 60 is failed (threshold=3) [............................XX.......X..X.......XXXXXXXX.XX.] duration:
19.477274844s
FAIL (2025-08-26 05:16:18 UTC)
operation: Commit uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp 10.37.0.83:6379: operation was canceled
Duration: 19.477303058s
core
uplink
us1
|