Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [............X.........X................X.........X..........] duration:
128.911172ms
FAIL (2025-08-17 05:23:26 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:15:28.257081331 (203h49m8.988348961s) [upload=212101]: Starting upload Aug 17 05:15:28.257138307 (203h49m8.988405928s) [upload=212101]: Done starting upload Aug 17 05:15:28.257217798 (203h49m8.988485418s) [upload=212101]: Uploading segments... Aug 17 05:15:28.257259375 (203h49m8.988527005s) [upload=212101]: Got next segment. Inline: true Aug 17 05:15:28.257265016 (203h49m8.988532646s) [upload=212101]: Waiting for error group managing segments... Aug 17 05:15:28.257303979 (203h49m8.988571609s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:15:28.257307786 (203h49m8.988575406s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:15:28.257311353 (203h49m8.988578983s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:15:28.260022307 (203h49m8.991289957s) [upload=212101]: Done uploading segments... Aug 17 05:15:28.260037555 (203h49m8.991305206s) [upload=212101]: Upload finished. err: metaclient: Too Many Requests
Duration: 128.936992ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [.............X.........X................X.........X.........] duration:
403.408375ms
FAIL (2025-08-17 05:22:50 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:15:28.257081331 (203h49m8.988348961s) [upload=212101]: Starting upload Aug 17 05:15:28.257138307 (203h49m8.988405928s) [upload=212101]: Done starting upload Aug 17 05:15:28.257217798 (203h49m8.988485418s) [upload=212101]: Uploading segments... Aug 17 05:15:28.257259375 (203h49m8.988527005s) [upload=212101]: Got next segment. Inline: true Aug 17 05:15:28.257265016 (203h49m8.988532646s) [upload=212101]: Waiting for error group managing segments... Aug 17 05:15:28.257303979 (203h49m8.988571609s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:15:28.257307786 (203h49m8.988575406s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:15:28.257311353 (203h49m8.988578983s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:15:28.260022307 (203h49m8.991289957s) [upload=212101]: Done uploading segments... Aug 17 05:15:28.260037555 (203h49m8.991305206s) [upload=212101]: Upload finished. err: metaclient: Too Many Requests
Duration: 403.425788ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
321.80757ms
FAIL (2025-08-17 05:22:32 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 321.873596ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [..............X.........X................X.........X........] duration:
164.477902ms
FAIL (2025-08-17 05:22:19 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:15:28.257081331 (203h49m8.988348961s) [upload=212101]: Starting upload Aug 17 05:15:28.257138307 (203h49m8.988405928s) [upload=212101]: Done starting upload Aug 17 05:15:28.257217798 (203h49m8.988485418s) [upload=212101]: Uploading segments... Aug 17 05:15:28.257259375 (203h49m8.988527005s) [upload=212101]: Got next segment. Inline: true Aug 17 05:15:28.257265016 (203h49m8.988532646s) [upload=212101]: Waiting for error group managing segments... Aug 17 05:15:28.257303979 (203h49m8.988571609s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:15:28.257307786 (203h49m8.988575406s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:15:28.257311353 (203h49m8.988578983s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:15:28.260022307 (203h49m8.991289957s) [upload=212101]: Done uploading segments... Aug 17 05:15:28.260037555 (203h49m8.991305206s) [upload=212101]: Upload finished. err: metaclient: Too Many Requests
Duration: 164.503881ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
419.553573ms
FAIL (2025-08-17 05:22:14 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 419.581976ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
422.454061ms
FAIL (2025-08-17 05:21:23 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 422.475952ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [...............X.........X................X.........X.......] duration:
166.862721ms
FAIL (2025-08-17 05:21:13 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:15:28.257081331 (203h49m8.988348961s) [upload=212101]: Starting upload Aug 17 05:15:28.257138307 (203h49m8.988405928s) [upload=212101]: Done starting upload Aug 17 05:15:28.257217798 (203h49m8.988485418s) [upload=212101]: Uploading segments... Aug 17 05:15:28.257259375 (203h49m8.988527005s) [upload=212101]: Got next segment. Inline: true Aug 17 05:15:28.257265016 (203h49m8.988532646s) [upload=212101]: Waiting for error group managing segments... Aug 17 05:15:28.257303979 (203h49m8.988571609s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:15:28.257307786 (203h49m8.988575406s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:15:28.257311353 (203h49m8.988578983s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:15:28.260022307 (203h49m8.991289957s) [upload=212101]: Done uploading segments... Aug 17 05:15:28.260037555 (203h49m8.991305206s) [upload=212101]: Upload finished. err: metaclient: Too Many Requests
Duration: 166.883661ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
586.551316ms
FAIL (2025-08-17 05:20:15 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 586.568339ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [................X.........X................X.........X......] duration:
370.119755ms
FAIL (2025-08-17 05:20:01 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:15:28.257081331 (203h49m8.988348961s) [upload=212101]: Starting upload Aug 17 05:15:28.257138307 (203h49m8.988405928s) [upload=212101]: Done starting upload Aug 17 05:15:28.257217798 (203h49m8.988485418s) [upload=212101]: Uploading segments... Aug 17 05:15:28.257259375 (203h49m8.988527005s) [upload=212101]: Got next segment. Inline: true Aug 17 05:15:28.257265016 (203h49m8.988532646s) [upload=212101]: Waiting for error group managing segments... Aug 17 05:15:28.257303979 (203h49m8.988571609s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:15:28.257307786 (203h49m8.988575406s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:15:28.257311353 (203h49m8.988578983s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:15:28.260022307 (203h49m8.991289957s) [upload=212101]: Done uploading segments... Aug 17 05:15:28.260037555 (203h49m8.991305206s) [upload=212101]: Upload finished. err: metaclient: Too Many Requests
Duration: 370.137347ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
374.019747ms
FAIL (2025-08-17 05:19:47 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 374.035837ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
664.646323ms
FAIL (2025-08-17 05:19:25 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 664.673005ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
494.083114ms
FAIL (2025-08-17 05:19:01 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 494.107109ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [.................X.........X................X.........X.....] duration:
165.353624ms
FAIL (2025-08-17 05:18:56 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:15:28.257081331 (203h49m8.988348961s) [upload=212101]: Starting upload Aug 17 05:15:28.257138307 (203h49m8.988405928s) [upload=212101]: Done starting upload Aug 17 05:15:28.257217798 (203h49m8.988485418s) [upload=212101]: Uploading segments... Aug 17 05:15:28.257259375 (203h49m8.988527005s) [upload=212101]: Got next segment. Inline: true Aug 17 05:15:28.257265016 (203h49m8.988532646s) [upload=212101]: Waiting for error group managing segments... Aug 17 05:15:28.257303979 (203h49m8.988571609s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:15:28.257307786 (203h49m8.988575406s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:15:28.257311353 (203h49m8.988578983s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:15:28.260022307 (203h49m8.991289957s) [upload=212101]: Done uploading segments... Aug 17 05:15:28.260037555 (203h49m8.991305206s) [upload=212101]: Upload finished. err: metaclient: Too Many Requests
Duration: 165.424317ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
351.771529ms
FAIL (2025-08-17 05:18:20 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 351.798168ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [..................X.........X................X.........X....] duration:
149.026327ms
FAIL (2025-08-17 05:18:13 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:15:28.257081331 (203h49m8.988348961s) [upload=212101]: Starting upload Aug 17 05:15:28.257138307 (203h49m8.988405928s) [upload=212101]: Done starting upload Aug 17 05:15:28.257217798 (203h49m8.988485418s) [upload=212101]: Uploading segments... Aug 17 05:15:28.257259375 (203h49m8.988527005s) [upload=212101]: Got next segment. Inline: true Aug 17 05:15:28.257265016 (203h49m8.988532646s) [upload=212101]: Waiting for error group managing segments... Aug 17 05:15:28.257303979 (203h49m8.988571609s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:15:28.257307786 (203h49m8.988575406s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:15:28.257311353 (203h49m8.988578983s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:15:28.260022307 (203h49m8.991289957s) [upload=212101]: Done uploading segments... Aug 17 05:15:28.260037555 (203h49m8.991305206s) [upload=212101]: Upload finished. err: metaclient: Too Many Requests
Duration: 149.051725ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
450.081569ms
FAIL (2025-08-17 05:17:44 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 450.106785ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [...................X.........X................X.........X...] duration:
156.696467ms
FAIL (2025-08-17 05:17:39 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:15:28.257081331 (203h49m8.988348961s) [upload=212101]: Starting upload Aug 17 05:15:28.257138307 (203h49m8.988405928s) [upload=212101]: Done starting upload Aug 17 05:15:28.257217798 (203h49m8.988485418s) [upload=212101]: Uploading segments... Aug 17 05:15:28.257259375 (203h49m8.988527005s) [upload=212101]: Got next segment. Inline: true Aug 17 05:15:28.257265016 (203h49m8.988532646s) [upload=212101]: Waiting for error group managing segments... Aug 17 05:15:28.257303979 (203h49m8.988571609s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:15:28.257307786 (203h49m8.988575406s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:15:28.257311353 (203h49m8.988578983s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:15:28.260022307 (203h49m8.991289957s) [upload=212101]: Done uploading segments... Aug 17 05:15:28.260037555 (203h49m8.991305206s) [upload=212101]: Upload finished. err: metaclient: Too Many Requests
Duration: 156.722616ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
386.687001ms
FAIL (2025-08-17 05:17:24 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 386.792ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [....................X.........X................X.........X..] duration:
131.89018ms
FAIL (2025-08-17 05:16:43 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:15:28.257081331 (203h49m8.988348961s) [upload=212101]: Starting upload Aug 17 05:15:28.257138307 (203h49m8.988405928s) [upload=212101]: Done starting upload Aug 17 05:15:28.257217798 (203h49m8.988485418s) [upload=212101]: Uploading segments... Aug 17 05:15:28.257259375 (203h49m8.988527005s) [upload=212101]: Got next segment. Inline: true Aug 17 05:15:28.257265016 (203h49m8.988532646s) [upload=212101]: Waiting for error group managing segments... Aug 17 05:15:28.257303979 (203h49m8.988571609s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:15:28.257307786 (203h49m8.988575406s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:15:28.257311353 (203h49m8.988578983s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:15:28.260022307 (203h49m8.991289957s) [upload=212101]: Done uploading segments... Aug 17 05:15:28.260037555 (203h49m8.991305206s) [upload=212101]: Upload finished. err: metaclient: Too Many Requests
Duration: 131.916841ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
399.460305ms
FAIL (2025-08-17 05:16:39 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 399.495573ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [.....................X.........X................X.........X.] duration:
132.651596ms
FAIL (2025-08-17 05:16:14 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:15:28.257081331 (203h49m8.988348961s) [upload=212101]: Starting upload Aug 17 05:15:28.257138307 (203h49m8.988405928s) [upload=212101]: Done starting upload Aug 17 05:15:28.257217798 (203h49m8.988485418s) [upload=212101]: Uploading segments... Aug 17 05:15:28.257259375 (203h49m8.988527005s) [upload=212101]: Got next segment. Inline: true Aug 17 05:15:28.257265016 (203h49m8.988532646s) [upload=212101]: Waiting for error group managing segments... Aug 17 05:15:28.257303979 (203h49m8.988571609s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:15:28.257307786 (203h49m8.988575406s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:15:28.257311353 (203h49m8.988578983s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:15:28.260022307 (203h49m8.991289957s) [upload=212101]: Done uploading segments... Aug 17 05:15:28.260037555 (203h49m8.991305206s) [upload=212101]: Upload finished. err: metaclient: Too Many Requests
Duration: 132.671082ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [......................X.........X................X.........X] duration:
16.181731ms
FAIL (2025-08-17 05:15:28 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:15:28.257081331 (203h49m8.988348961s) [upload=212101]: Starting upload Aug 17 05:15:28.257138307 (203h49m8.988405928s) [upload=212101]: Done starting upload Aug 17 05:15:28.257217798 (203h49m8.988485418s) [upload=212101]: Uploading segments... Aug 17 05:15:28.257259375 (203h49m8.988527005s) [upload=212101]: Got next segment. Inline: true Aug 17 05:15:28.257265016 (203h49m8.988532646s) [upload=212101]: Waiting for error group managing segments... Aug 17 05:15:28.257303979 (203h49m8.988571609s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:15:28.257307786 (203h49m8.988575406s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:15:28.257311353 (203h49m8.988578983s) [upload=212101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:15:28.260022307 (203h49m8.991289957s) [upload=212101]: Done uploading segments... Aug 17 05:15:28.260037555 (203h49m8.991305206s) [upload=212101]: Upload finished. err: metaclient: Too Many Requests
Duration: 16.197791ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
415.90405ms
FAIL (2025-08-17 05:15:27 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 415.927153ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
294.873972ms
FAIL (2025-08-17 05:15:11 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 294.896174ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
3 out of 60 is failed (threshold=3) [.......................X.........X................X.........] duration:
344.381678ms
FAIL (2025-08-17 05:14:36 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:07:35.778610178 (203h41m16.509877828s) [upload=211959]: Starting upload Aug 17 05:07:35.778686041 (203h41m16.509953691s) [upload=211959]: Done starting upload Aug 17 05:07:35.778757214 (203h41m16.510024865s) [upload=211959]: Uploading segments... Aug 17 05:07:35.778802471 (203h41m16.510070121s) [upload=211959]: Got next segment. Inline: true Aug 17 05:07:35.778808221 (203h41m16.510075872s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:07:35.778851903 (203h41m16.510119554s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:07:35.778857173 (203h41m16.510124823s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:07:35.778860800 (203h41m16.51012845s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:07:35.781460574 (203h41m16.512728194s) [upload=211959]: Done uploading segments... Aug 17 05:07:35.781471214 (203h41m16.512738833s) [upload=211959]: Upload finished. err: metaclient: Too Many Requests
Duration: 344.397868ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
372.227693ms
FAIL (2025-08-17 05:14:23 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 372.257068ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
355.914615ms
FAIL (2025-08-17 05:13:50 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 355.943099ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
3 out of 60 is failed (threshold=3) [........................X.........X................X........] duration:
156.798386ms
FAIL (2025-08-17 05:13:27 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:07:35.778610178 (203h41m16.509877828s) [upload=211959]: Starting upload Aug 17 05:07:35.778686041 (203h41m16.509953691s) [upload=211959]: Done starting upload Aug 17 05:07:35.778757214 (203h41m16.510024865s) [upload=211959]: Uploading segments... Aug 17 05:07:35.778802471 (203h41m16.510070121s) [upload=211959]: Got next segment. Inline: true Aug 17 05:07:35.778808221 (203h41m16.510075872s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:07:35.778851903 (203h41m16.510119554s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:07:35.778857173 (203h41m16.510124823s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:07:35.778860800 (203h41m16.51012845s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:07:35.781460574 (203h41m16.512728194s) [upload=211959]: Done uploading segments... Aug 17 05:07:35.781471214 (203h41m16.512738833s) [upload=211959]: Upload finished. err: metaclient: Too Many Requests
Duration: 156.853741ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
364.461856ms
FAIL (2025-08-17 05:13:07 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 364.560451ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
468.677041ms
FAIL (2025-08-17 05:12:17 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 468.704763ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
3 out of 60 is failed (threshold=3) [.........................X.........X................X.......] duration:
165.813831ms
FAIL (2025-08-17 05:12:16 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:07:35.778610178 (203h41m16.509877828s) [upload=211959]: Starting upload Aug 17 05:07:35.778686041 (203h41m16.509953691s) [upload=211959]: Done starting upload Aug 17 05:07:35.778757214 (203h41m16.510024865s) [upload=211959]: Uploading segments... Aug 17 05:07:35.778802471 (203h41m16.510070121s) [upload=211959]: Got next segment. Inline: true Aug 17 05:07:35.778808221 (203h41m16.510075872s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:07:35.778851903 (203h41m16.510119554s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:07:35.778857173 (203h41m16.510124823s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:07:35.778860800 (203h41m16.51012845s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:07:35.781460574 (203h41m16.512728194s) [upload=211959]: Done uploading segments... Aug 17 05:07:35.781471214 (203h41m16.512738833s) [upload=211959]: Upload finished. err: metaclient: Too Many Requests
Duration: 165.831785ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
373.165558ms
FAIL (2025-08-17 05:11:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 373.184384ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
344.346149ms
FAIL (2025-08-17 05:11:32 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 344.369281ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
3 out of 60 is failed (threshold=3) [..........................X.........X................X......] duration:
161.385874ms
FAIL (2025-08-17 05:11:16 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:07:35.778610178 (203h41m16.509877828s) [upload=211959]: Starting upload Aug 17 05:07:35.778686041 (203h41m16.509953691s) [upload=211959]: Done starting upload Aug 17 05:07:35.778757214 (203h41m16.510024865s) [upload=211959]: Uploading segments... Aug 17 05:07:35.778802471 (203h41m16.510070121s) [upload=211959]: Got next segment. Inline: true Aug 17 05:07:35.778808221 (203h41m16.510075872s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:07:35.778851903 (203h41m16.510119554s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:07:35.778857173 (203h41m16.510124823s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:07:35.778860800 (203h41m16.51012845s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:07:35.781460574 (203h41m16.512728194s) [upload=211959]: Done uploading segments... Aug 17 05:07:35.781471214 (203h41m16.512738833s) [upload=211959]: Upload finished. err: metaclient: Too Many Requests
Duration: 161.409108ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
447.914937ms
FAIL (2025-08-17 05:11:03 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 447.942288ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
3 out of 60 is failed (threshold=3) [...........................X.........X................X.....] duration:
403.693445ms
FAIL (2025-08-17 05:10:55 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:07:35.778610178 (203h41m16.509877828s) [upload=211959]: Starting upload Aug 17 05:07:35.778686041 (203h41m16.509953691s) [upload=211959]: Done starting upload Aug 17 05:07:35.778757214 (203h41m16.510024865s) [upload=211959]: Uploading segments... Aug 17 05:07:35.778802471 (203h41m16.510070121s) [upload=211959]: Got next segment. Inline: true Aug 17 05:07:35.778808221 (203h41m16.510075872s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:07:35.778851903 (203h41m16.510119554s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:07:35.778857173 (203h41m16.510124823s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:07:35.778860800 (203h41m16.51012845s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:07:35.781460574 (203h41m16.512728194s) [upload=211959]: Done uploading segments... Aug 17 05:07:35.781471214 (203h41m16.512738833s) [upload=211959]: Upload finished. err: metaclient: Too Many Requests
Duration: 403.71706ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
421.835756ms
FAIL (2025-08-17 05:10:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 421.854911ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
356.37427ms
FAIL (2025-08-17 05:10:30 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 356.396512ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
3 out of 60 is failed (threshold=3) [............................X.........X................X....] duration:
128.952089ms
FAIL (2025-08-17 05:10:17 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:07:35.778610178 (203h41m16.509877828s) [upload=211959]: Starting upload Aug 17 05:07:35.778686041 (203h41m16.509953691s) [upload=211959]: Done starting upload Aug 17 05:07:35.778757214 (203h41m16.510024865s) [upload=211959]: Uploading segments... Aug 17 05:07:35.778802471 (203h41m16.510070121s) [upload=211959]: Got next segment. Inline: true Aug 17 05:07:35.778808221 (203h41m16.510075872s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:07:35.778851903 (203h41m16.510119554s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:07:35.778857173 (203h41m16.510124823s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:07:35.778860800 (203h41m16.51012845s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:07:35.781460574 (203h41m16.512728194s) [upload=211959]: Done uploading segments... Aug 17 05:07:35.781471214 (203h41m16.512738833s) [upload=211959]: Upload finished. err: metaclient: Too Many Requests
Duration: 128.974122ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
654.843107ms
FAIL (2025-08-17 05:09:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 654.871691ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
3 out of 60 is failed (threshold=3) [.............................X.........X................X...] duration:
198.435385ms
FAIL (2025-08-17 05:09:37 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:07:35.778610178 (203h41m16.509877828s) [upload=211959]: Starting upload Aug 17 05:07:35.778686041 (203h41m16.509953691s) [upload=211959]: Done starting upload Aug 17 05:07:35.778757214 (203h41m16.510024865s) [upload=211959]: Uploading segments... Aug 17 05:07:35.778802471 (203h41m16.510070121s) [upload=211959]: Got next segment. Inline: true Aug 17 05:07:35.778808221 (203h41m16.510075872s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:07:35.778851903 (203h41m16.510119554s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:07:35.778857173 (203h41m16.510124823s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:07:35.778860800 (203h41m16.51012845s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:07:35.781460574 (203h41m16.512728194s) [upload=211959]: Done uploading segments... Aug 17 05:07:35.781471214 (203h41m16.512738833s) [upload=211959]: Upload finished. err: metaclient: Too Many Requests
Duration: 198.470972ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
566.947664ms
FAIL (2025-08-17 05:09:36 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 566.971989ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
3 out of 60 is failed (threshold=3) [..............................X.........X................X..] duration:
142.793075ms
FAIL (2025-08-17 05:09:04 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:07:35.778610178 (203h41m16.509877828s) [upload=211959]: Starting upload Aug 17 05:07:35.778686041 (203h41m16.509953691s) [upload=211959]: Done starting upload Aug 17 05:07:35.778757214 (203h41m16.510024865s) [upload=211959]: Uploading segments... Aug 17 05:07:35.778802471 (203h41m16.510070121s) [upload=211959]: Got next segment. Inline: true Aug 17 05:07:35.778808221 (203h41m16.510075872s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:07:35.778851903 (203h41m16.510119554s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:07:35.778857173 (203h41m16.510124823s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:07:35.778860800 (203h41m16.51012845s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:07:35.781460574 (203h41m16.512728194s) [upload=211959]: Done uploading segments... Aug 17 05:07:35.781471214 (203h41m16.512738833s) [upload=211959]: Upload finished. err: metaclient: Too Many Requests
Duration: 142.810748ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
380.354119ms
FAIL (2025-08-17 05:08:55 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 380.38158ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
416.003058ms
FAIL (2025-08-17 05:08:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
se
Aug 17 04:23:28.620840633 (202h57m9.352108284s) [upload=211175]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 17 04:23:28.620845953 (202h57m9.352113603s) [upload=211175]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 17 04:23:28.646835158 (202h57m9.378102858s) [seg_pos={0 0}, upload=211175]: Begin upload segment...
Aug 17 04:23:28.646936819 (202h57m9.378204468s) [seg_pos={0 0}, upload=211175]: Done begin upload segment.
Aug 17 04:23:28.646989559 (202h57m9.378257218s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.647051715 (202h57m9.378319375s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.647662535 (202h57m9.378930186s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.647827836 (202h57m9.379095487s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.648134293 (202h57m9.379401954s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.648314913 (202h57m9.379582573s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.648543414 (202h57m9.379811063s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.648834572 (202h57m9.380102222s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.648992249 (202h57m9.380259908s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.649335063 (202h57m9.380602713s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.649507127 (202h57m9.380774777s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.649830226 (202h57m9.381097876s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.650003341 (202h57m9.381270992s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.650256469 (202h57m9.381524128s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.650434553 (202h57m9.381702213s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.650709621 (202h57m9.381977282s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.650857891 (202h57m9.38212554s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.651270246 (202h57m9.382537897s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.768946979 (202h57m9.500214629s) [upload=211175]: Next returned nil segment
Aug 17 04:23:28.768974191 (202h57m9.500241841s) [upload=211175]: Waiting for error group managing segments...
Aug 17 04:23:28.777572686 (202h57m9.508840347s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.778945461 (202h57m9.510213112s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.785896566 (202h57m9.517164206s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.793497595 (202h57m9.524765245s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.796220951 (202h57m9.527488592s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.799941495 (202h57m9.531209135s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.801171361 (202h57m9.532439001s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.802452863 (202h57m9.533720513s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.822657288 (202h57m9.553924938s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.874167434 (202h57m9.605435084s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:29.100609905 (202h57m9.831877555s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:29.216981541 (202h57m9.948249171s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:24:28.606052339 (202h58m9.337319999s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 17 04:24:28.606141437 (202h58m9.337409087s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1AowWwVvxzmnW5qwbqbMBVbvNoigffjAN51LQp8e6XA9CZkeXL, address:104.36.82.3:4790): protocol: context deadline exceeded; context deadline exceeded
Aug 17 04:24:28.606261834 (202h58m9.337529484s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:58906->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 17 04:24:28.606338818 (202h58m9.337606469s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:104.36.82.23:4455): protocol: write tcp 5.161.89.252:44290->104.36.82.23:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 17 04:24:28.606369045 (202h58m9.337636696s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 17 04:24:28.606443665 (202h58m9.337711316s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, address:104.36.82.49:4003): protocol: context deadline exceeded; context deadline exceeded
Aug 17 04:24:28.606480044 (202h58m9.337747704s) [seg_pos={0 0}, upload=211175]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Aug 17 04:24:28.621101719 (202h58m9.352369369s) [upload=211175]: Done uploading segments...
Aug 17 04:24:28.621117609 (202h58m9.35238526s) [upload=211175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 416.026011ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X.XX........................................................] duration:
378.052598ms
FAIL (2025-08-17 05:08:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 378.150162ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
633.140405ms
FAIL (2025-08-17 05:08:25 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:42:45.449447831 (203h16m26.180715511s) [upload=211511]: Starting upload Aug 17 04:42:45.449550555 (203h16m26.180818204s) [upload=211511]: Done starting upload Aug 17 04:42:45.449707489 (203h16m26.180975231s) [upload=211511]: Uploading segments... Aug 17 04:42:45.449993998 (203h16m26.181261649s) [upload=211511]: Got next segment. Inline: false Aug 17 04:42:45.450054783 (203h16m26.181322433s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 04:42:45.450060414 (203h16m26.181328064s) [upload=211511]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 17 04:42:45.535432247 (203h16m26.266699898s) [upload=211511]: Next segment err: metaclient: Too Many Requests Aug 17 04:42:45.535462685 (203h16m26.266730335s) [upload=211511]: Done uploading segments... Aug 17 04:42:45.535483674 (203h16m26.266751334s) [upload=211511]: Upload finished. err: metaclient: Too Many Requests
Duration: 633.162618ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
3 out of 60 is failed (threshold=3) [...............................X.........X................X.] duration:
176.07858ms
FAIL (2025-08-17 05:08:18 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 05:07:35.778610178 (203h41m16.509877828s) [upload=211959]: Starting upload Aug 17 05:07:35.778686041 (203h41m16.509953691s) [upload=211959]: Done starting upload Aug 17 05:07:35.778757214 (203h41m16.510024865s) [upload=211959]: Uploading segments... Aug 17 05:07:35.778802471 (203h41m16.510070121s) [upload=211959]: Got next segment. Inline: true Aug 17 05:07:35.778808221 (203h41m16.510075872s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:07:35.778851903 (203h41m16.510119554s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 17 05:07:35.778857173 (203h41m16.510124823s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 17 05:07:35.778860800 (203h41m16.51012845s) [upload=211959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 17 05:07:35.781460574 (203h41m16.512728194s) [upload=211959]: Done uploading segments... Aug 17 05:07:35.781471214 (203h41m16.512738833s) [upload=211959]: Upload finished. err: metaclient: Too Many Requests
Duration: 176.093619ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
483.551831ms
FAIL (2025-08-17 05:08:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
se
Aug 17 04:23:28.620840633 (202h57m9.352108284s) [upload=211175]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 17 04:23:28.620845953 (202h57m9.352113603s) [upload=211175]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 17 04:23:28.646835158 (202h57m9.378102858s) [seg_pos={0 0}, upload=211175]: Begin upload segment...
Aug 17 04:23:28.646936819 (202h57m9.378204468s) [seg_pos={0 0}, upload=211175]: Done begin upload segment.
Aug 17 04:23:28.646989559 (202h57m9.378257218s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.647051715 (202h57m9.378319375s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.647662535 (202h57m9.378930186s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.647827836 (202h57m9.379095487s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.648134293 (202h57m9.379401954s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.648314913 (202h57m9.379582573s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.648543414 (202h57m9.379811063s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.648834572 (202h57m9.380102222s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.648992249 (202h57m9.380259908s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.649335063 (202h57m9.380602713s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.649507127 (202h57m9.380774777s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.649830226 (202h57m9.381097876s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.650003341 (202h57m9.381270992s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.650256469 (202h57m9.381524128s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.650434553 (202h57m9.381702213s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.650709621 (202h57m9.381977282s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.650857891 (202h57m9.38212554s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.651270246 (202h57m9.382537897s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Uploading piece...
Aug 17 04:23:28.768946979 (202h57m9.500214629s) [upload=211175]: Next returned nil segment
Aug 17 04:23:28.768974191 (202h57m9.500241841s) [upload=211175]: Waiting for error group managing segments...
Aug 17 04:23:28.777572686 (202h57m9.508840347s) [noise=true , address=77.90.31.11:4558 , piece_id=EQVKQ7VF476HIAJ2RS5KOLHKC2G3IGFAOD4EDB3SSVNYXXXXOKSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.778945461 (202h57m9.510213112s) [noise=true , address=45.91.116.14:4210 , piece_id=B5WJCBD7TMMJV2UZOB2H7PTMOUQU3IKFQULK7RR6GM2YXNHGXTWQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.785896566 (202h57m9.517164206s) [noise=true , address=45.91.116.6:4976 , piece_id=I7VRMIIQOQH5FTN5KXIDKOHRSLOOWWMIETXVALFZ2TMWYY5RIEFA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.793497595 (202h57m9.524765245s) [noise=true , address=45.91.116.8:4075 , piece_id=6MXDPEQ5STD2NJFMCZGUE3CJPKG7QV7XLVGIMJTF3AL5TCCUW5QQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.796220951 (202h57m9.527488592s) [noise=true , address=77.90.31.13:4647 , piece_id=NFEC7SXBDQORDGZA2AAZIEPWVC4BSS6ZYVEPF4NG2VPHFWOX2KJQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.799941495 (202h57m9.531209135s) [noise=true , address=45.91.116.12:4242 , piece_id=2YFHC6NRUPE6S6A6ZEZIFT6WVWPNHNNZJI22DUSX6SIKXOATGHNQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.801171361 (202h57m9.532439001s) [noise=true , address=77.90.31.18:4810 , piece_id=KKU7XFAW572XIR656DOASOGTTNXXHUCDQM4CP6DLCHMJTGH53SXA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.802452863 (202h57m9.533720513s) [noise=true , address=45.91.116.13:4172 , piece_id=NTZMLMVOOUNTLTI2AIJFRVJ5FWESBIXFF7IFZEAESDOY74V3K24A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.822657288 (202h57m9.553924938s) [noise=true , address=45.91.116.7:4035 , piece_id=UJOLVS7QCIOJI7VGOUSZEEF5QFKTAEY5EOOR6LTLUI5XKITDJLIA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:28.874167434 (202h57m9.605435084s) [noise=true , address=77.90.31.19:4853 , piece_id=BKG3LAQW65IFWIOIINQ7AK3TFISFYVQXWH55LLWSHZGGHYYBSSDA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:29.100609905 (202h57m9.831877555s) [noise=true , address=79.127.201.215:4345 , piece_id=NOSCVVA3X5XJ37XUXABNBAVLNCZLS5EAE2RP7SRT2P7L6A2CKBZQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:23:29.216981541 (202h57m9.948249171s) [noise=true , address=79.127.205.246:4288 , piece_id=76LPFGIPIVE3MXAZVAROZSCMEWZWMQDEBKLF77TTJPOOVDLZ7AAQ, seg_pos={0 0}, upload=211175]: Done uploading piece. err: <nil>
Aug 17 04:24:28.606052339 (202h58m9.337319999s) [noise=true , address=104.36.82.44:4886 , piece_id=IDBWK46PJZRER44U4RB6LUQAGTZEDMSOPAVRBH2CV56RM3JS3I3A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 17 04:24:28.606141437 (202h58m9.337409087s) [noise=true , address=104.36.82.3:4790 , piece_id=OKZYIEHAI4GGPWYB5SF3JYN7CFCOU6S66JOLC3ULFJBBYHAPMFCA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1AowWwVvxzmnW5qwbqbMBVbvNoigffjAN51LQp8e6XA9CZkeXL, address:104.36.82.3:4790): protocol: context deadline exceeded; context deadline exceeded
Aug 17 04:24:28.606261834 (202h58m9.337529484s) [noise=true , address=104.36.82.9:4025 , piece_id=5UFGQ5VECYEBSBA4CAWQNICGM4CY2WBMIKIWMBE3YIGKZVBO7I6A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:58906->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 17 04:24:28.606338818 (202h58m9.337606469s) [noise=true , address=104.36.82.23:4455 , piece_id=BCSYWGESJWL56J63GIUEIV3BRWOO4YRW4UXUMCGB37J53CBMUVSA, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:104.36.82.23:4455): protocol: write tcp 5.161.89.252:44290->104.36.82.23:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 17 04:24:28.606369045 (202h58m9.337636696s) [noise=true , address=104.36.82.60:4365 , piece_id=X2UBI33FNGXGDXW7JFVUYJ4WMQQVEH2P5W2J2NZ6TNDNP6DJSM2A, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 17 04:24:28.606443665 (202h58m9.337711316s) [noise=true , address=104.36.82.49:4003 , piece_id=HTLAF4GFG7OBKTKFYLXF5QMOAPWAEL7ZD2IJ64BO5NRJGEVKIN5Q, seg_pos={0 0}, upload=211175]: Done uploading piece. err: ecclient: upload failed (node:1A8MjPSMJHMxQqBbrjewwyZ4MKTwpvYHh3CE7rP3B9NeYgdfmW, address:104.36.82.49:4003): protocol: context deadline exceeded; context deadline exceeded
Aug 17 04:24:28.606480044 (202h58m9.337747704s) [seg_pos={0 0}, upload=211175]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Aug 17 04:24:28.621101719 (202h58m9.352369369s) [upload=211175]: Done uploading segments...
Aug 17 04:24:28.621117609 (202h58m9.35238526s) [upload=211175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 483.57208ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X.XX.......................................................] duration:
413.568867ms
FAIL (2025-08-17 05:08:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 413.585928ms
edge
gateway
us-select-1
|