Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.287115895s
FAIL (2025-08-18 19:44:22 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 5.287140982s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.........................X...............X..................] duration:
398.084005ms
FAIL (2025-08-18 19:43:53 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 398.104244ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X....X.....................] duration:
492.422504ms
FAIL (2025-08-18 19:43:44 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 492.447671ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
4.937957876s
FAIL (2025-08-18 19:43:39 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.93800712s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................X....X....................] duration:
463.686845ms
FAIL (2025-08-18 19:43:18 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 463.70558ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [..........................X...............X.................] duration:
288.342826ms
FAIL (2025-08-18 19:43:13 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 288.360059ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.030609569s
FAIL (2025-08-18 19:42:52 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 5.030633664s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X....X...................] duration:
720.585825ms
FAIL (2025-08-18 19:42:36 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 720.621603ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [...........................X...............X................] duration:
164.562763ms
FAIL (2025-08-18 19:42:28 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 164.578343ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
5.29782855s
FAIL (2025-08-18 19:42:24 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 5.2978494s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
4.91720905s
FAIL (2025-08-18 19:42:01 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.917232834s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [............................X...............X...............] duration:
148.247141ms
FAIL (2025-08-18 19:41:39 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 148.26759ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X....X..................] duration:
457.793385ms
FAIL (2025-08-18 19:41:38 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 457.817079ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.............................X...............X..............] duration:
214.607918ms
FAIL (2025-08-18 19:41:13 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 214.626732ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
5.25356872s
FAIL (2025-08-18 19:41:06 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 5.253596262s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X....X.................] duration:
457.24358ms
FAIL (2025-08-18 19:40:31 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 457.265181ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
4.904177741s
FAIL (2025-08-18 19:40:15 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.904205113s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [..............................X...............X.............] duration:
114.271818ms
FAIL (2025-08-18 19:40:06 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 114.291114ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
4.919372872s
FAIL (2025-08-18 19:39:33 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.919397849s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................X....X................] duration:
485.165959ms
FAIL (2025-08-18 19:39:32 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 485.219441ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X....X...............] duration:
816.424825ms
FAIL (2025-08-18 19:39:16 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 816.447848ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [...............................X...............X............] duration:
170.967917ms
FAIL (2025-08-18 19:39:00 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 170.982265ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
4.954999297s
FAIL (2025-08-18 19:38:49 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.955018293s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X....X..............] duration:
497.767183ms
FAIL (2025-08-18 19:38:40 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 497.794545ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [................................X...............X...........] duration:
133.872794ms
FAIL (2025-08-18 19:38:29 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 133.893112ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X....X.............] duration:
734.569789ms
FAIL (2025-08-18 19:38:14 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 734.596258ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
5.062365831s
FAIL (2025-08-18 19:37:47 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 5.062400606s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.................................X...............X..........] duration:
133.941985ms
FAIL (2025-08-18 19:37:44 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 133.972813ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X....X............] duration:
542.472967ms
FAIL (2025-08-18 19:37:42 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 542.499726ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [..................................X...............X.........] duration:
223.725097ms
FAIL (2025-08-18 19:37:27 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 223.748881ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
5.187896112s
FAIL (2025-08-18 19:37:12 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 5.187923704s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X....X...........] duration:
516.374743ms
FAIL (2025-08-18 19:36:59 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 516.397445ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [...................................X...............X........] duration:
151.766519ms
FAIL (2025-08-18 19:36:54 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 151.789031ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X....X..........] duration:
469.761729ms
FAIL (2025-08-18 19:36:31 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 469.776708ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
4.887958332s
FAIL (2025-08-18 19:36:22 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.887984891s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [....................................X...............X.......] duration:
221.674772ms
FAIL (2025-08-18 19:36:07 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 221.704026ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X....X.........] duration:
832.592148ms
FAIL (2025-08-18 19:35:51 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 832.613478ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X...............X......] duration:
266.220041ms
FAIL (2025-08-18 19:35:46 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 266.242513ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
5.351758312s
FAIL (2025-08-18 19:35:44 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 5.35178449s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [......................................X...............X.....] duration:
142.515139ms
FAIL (2025-08-18 19:35:28 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 142.540186ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
4.884533645s
FAIL (2025-08-18 19:35:07 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.884588809s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X....X........] duration:
421.868695ms
FAIL (2025-08-18 19:34:54 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 421.892941ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X...............X....] duration:
183.973755ms
FAIL (2025-08-18 19:34:35 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 183.992791ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
5.433085809s
FAIL (2025-08-18 19:34:19 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 5.433111697s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [........................................X...............X...] duration:
162.234029ms
FAIL (2025-08-18 19:33:57 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 162.255551ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X....X.......] duration:
559.098515ms
FAIL (2025-08-18 19:33:54 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 559.118303ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
4.909017998s
FAIL (2025-08-18 19:33:45 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.90904005s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X...............X..] duration:
182.523913ms
FAIL (2025-08-18 19:33:14 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:32:04.150795103 (242h5m44.882062752s) [upload=251816]: Starting upload Aug 18 19:32:04.150971773 (242h5m44.882239424s) [upload=251816]: Done starting upload Aug 18 19:32:04.151100085 (242h5m44.882367746s) [upload=251816]: Uploading segments... Aug 18 19:32:04.151159036 (242h5m44.882426696s) [upload=251816]: Got next segment. Inline: true Aug 18 19:32:04.151166620 (242h5m44.88243427s) [upload=251816]: Waiting for error group managing segments... Aug 18 19:32:04.151227063 (242h5m44.882494723s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:32:04.151235009 (242h5m44.882502658s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 18 19:32:04.151240087 (242h5m44.882507738s) [upload=251816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 18 19:32:04.153992839 (242h5m44.885260499s) [upload=251816]: Done uploading segments... Aug 18 19:32:04.154012065 (242h5m44.885279715s) [upload=251816]: Upload finished. err: metaclient: Too Many Requests
Duration: 182.547326ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
4.980691806s
FAIL (2025-08-18 19:32:55 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.980719468s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X....X......] duration:
405.049871ms
FAIL (2025-08-18 19:32:44 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 18 19:27:32.511093581 (242h1m13.242361211s) [upload=251742]: Starting upload Aug 18 19:27:32.511142133 (242h1m13.242409763s) [upload=251742]: Done starting upload Aug 18 19:27:32.511287967 (242h1m13.242555597s) [upload=251742]: Uploading segments... Aug 18 19:27:32.511379579 (242h1m13.242647209s) [upload=251742]: Got next segment. Inline: false Aug 18 19:27:32.511430944 (242h1m13.242698565s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 18 19:27:32.511437096 (242h1m13.242704726s) [upload=251742]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 18 19:27:32.514566214 (242h1m13.245833844s) [upload=251742]: Next segment err: metaclient: Too Many Requests Aug 18 19:27:32.514576373 (242h1m13.245844003s) [upload=251742]: Done uploading segments... Aug 18 19:27:32.514585009 (242h1m13.245852639s) [upload=251742]: Upload finished. err: metaclient: Too Many Requests
Duration: 405.073998ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|