Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.193059833s
FAIL (2025-08-25 01:06:40 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.193085491s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
286.529702ms
FAIL (2025-08-25 01:06:27 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 286.558635ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
570.795886ms
FAIL (2025-08-25 01:05:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 570.811615ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
283.761417ms
FAIL (2025-08-25 01:05:30 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 283.781945ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
1.309604019s
FAIL (2025-08-25 01:05:27 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.309631399s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
432.107548ms
FAIL (2025-08-25 01:04:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 432.123558ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
423.913175ms
FAIL (2025-08-25 01:04:28 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 423.93727ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
1.753935687s
FAIL (2025-08-25 01:04:18 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.753962677s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
391.282183ms
FAIL (2025-08-25 01:04:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 391.304565ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
389.078152ms
FAIL (2025-08-25 01:03:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 389.100945ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
158.413618ms
FAIL (2025-08-25 01:03:45 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 158.439307ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
325.325599ms
FAIL (2025-08-25 01:03:18 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 325.345737ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
630.629873ms
FAIL (2025-08-25 01:03:17 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 630.649681ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
375.366803ms
FAIL (2025-08-25 01:02:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 375.382984ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
1.598161588s
FAIL (2025-08-25 01:02:39 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.598184912s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
190.560233ms
FAIL (2025-08-25 01:02:30 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 190.57999ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
361.400716ms
FAIL (2025-08-25 01:02:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 361.419281ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
582.10025ms
FAIL (2025-08-25 01:01:42 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 582.124636ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
510.5242ms
FAIL (2025-08-25 01:01:37 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 510.548455ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
620.355522ms
FAIL (2025-08-25 01:01:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 620.532936ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
3.851902796s
FAIL (2025-08-25 01:01:12 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 3.851922963s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
2.602817663s
FAIL (2025-08-25 01:01:11 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.602843512s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
228.959666ms
FAIL (2025-08-25 01:00:50 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 228.980205ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
494.67013ms
FAIL (2025-08-25 01:00:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 494.69093ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
1.991991378s
FAIL (2025-08-25 01:00:35 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.992010905s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
455.486483ms
FAIL (2025-08-25 01:00:16 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 455.5167ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
216.092395ms
FAIL (2025-08-25 01:00:09 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 216.130719ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
3.473357674s
FAIL (2025-08-25 01:00:08 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 3.473383883s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
374.332818ms
FAIL (2025-08-25 01:00:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 374.391178ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
364.65427ms
FAIL (2025-08-25 00:59:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 364.677433ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
162.297729ms
FAIL (2025-08-25 00:59:38 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 162.324509ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
1.821743251s
FAIL (2025-08-25 00:59:34 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.821765794s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
3.052875244s
FAIL (2025-08-25 00:59:02 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 3.05290071s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
300.393725ms
FAIL (2025-08-25 00:59:01 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 300.420014ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
1.366122203s
FAIL (2025-08-25 00:58:35 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.366153161s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
638.520833ms
FAIL (2025-08-25 00:58:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 638.639547ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
144.294334ms
FAIL (2025-08-25 00:58:26 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 144.315143ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..XX........................................................] duration:
3.280760442s
FAIL (2025-08-25 00:58:11 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 3.280782011s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
618.009547ms
FAIL (2025-08-25 00:57:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 618.037289ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
207.82563ms
FAIL (2025-08-25 00:57:39 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 207.843542ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
2.356399578s
FAIL (2025-08-25 00:57:34 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.356429494s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...XX.......................................................] duration:
3.192655967s
FAIL (2025-08-25 00:57:20 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 3.192682667s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
2.064645059s
FAIL (2025-08-25 00:57:07 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.064795883s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
500.304209ms
FAIL (2025-08-25 00:56:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 500.324677ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
350.856641ms
FAIL (2025-08-25 00:56:38 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 350.878752ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
137.252852ms
FAIL (2025-08-25 00:56:36 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 137.277338ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
383.837751ms
FAIL (2025-08-25 00:56:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 383.871856ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....XX......................................................] duration:
3.139401534s
FAIL (2025-08-25 00:56:19 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 3.139426381s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
1.768796675s
FAIL (2025-08-25 00:56:08 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:25:28.400683968 (390h59m9.131951598s) [upload=406444]: Starting upload Aug 25 00:25:28.400740554 (390h59m9.132008184s) [upload=406444]: Done starting upload Aug 25 00:25:28.400969294 (390h59m9.132236924s) [upload=406444]: Uploading segments... Aug 25 00:25:28.400998699 (390h59m9.132266329s) [upload=406444]: Got next segment. Inline: false Aug 25 00:25:28.401020590 (390h59m9.13228822s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:25:28.401024117 (390h59m9.132291747s) [upload=406444]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 25 00:25:28.403715205 (390h59m9.134982825s) [upload=406444]: Next segment err: metaclient: Too Many Requests Aug 25 00:25:28.403727398 (390h59m9.134995017s) [upload=406444]: Done uploading segments... Aug 25 00:25:28.403753247 (390h59m9.135020877s) [upload=406444]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.768825099s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
122.269142ms
FAIL (2025-08-25 00:56:04 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 25 00:34:31.724244888 (391h8m12.455512539s) [upload=406603]: Starting upload Aug 25 00:34:31.724305743 (391h8m12.455573392s) [upload=406603]: Done starting upload Aug 25 00:34:31.724386896 (391h8m12.455654555s) [upload=406603]: Uploading segments... Aug 25 00:34:31.724479178 (391h8m12.455746829s) [upload=406603]: Got next segment. Inline: true Aug 25 00:34:31.724484819 (391h8m12.455752479s) [upload=406603]: Waiting for error group managing segments... Aug 25 00:34:31.724528791 (391h8m12.455796442s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 25 00:34:31.724533932 (391h8m12.455801581s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 25 00:34:31.724537368 (391h8m12.455805018s) [upload=406603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 25 00:34:31.727506599 (391h8m12.458774249s) [upload=406603]: Done uploading segments... Aug 25 00:34:31.727519223 (391h8m12.458786872s) [upload=406603]: Upload finished. err: metaclient: Too Many Requests
Duration: 122.295492ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|