Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [....................XX......................................] duration:
187.676695ms
FAIL (2025-08-11 22:32:21 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 187.706913ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X...........X............................................] duration:
1.197425136s
FAIL (2025-08-11 22:32:08 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.197454171s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration:
257.399779ms
FAIL (2025-08-11 22:31:55 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 257.42598ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X...........X...........................................] duration:
648.821247ms
FAIL (2025-08-11 22:31:51 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 648.848248ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X...........X..........................................] duration:
703.43561ms
FAIL (2025-08-11 22:31:26 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 703.458744ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [......................XX....................................] duration:
303.201357ms
FAIL (2025-08-11 22:31:12 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 303.224931ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.......................XX...................................] duration:
287.081601ms
FAIL (2025-08-11 22:30:40 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 287.125013ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X...........X.........................................] duration:
789.209504ms
FAIL (2025-08-11 22:30:21 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 789.230083ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [........................XX..................................] duration:
416.56767ms
FAIL (2025-08-11 22:29:29 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 416.592987ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X...........X........................................] duration:
685.142488ms
FAIL (2025-08-11 22:29:27 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 686.902449ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.........................XX.................................] duration:
222.542189ms
FAIL (2025-08-11 22:28:47 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 222.560334ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X...........X.......................................] duration:
602.308619ms
FAIL (2025-08-11 22:28:13 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 602.335009ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [..........................XX................................] duration:
354.647027ms
FAIL (2025-08-11 22:27:52 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 354.673837ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X...........X......................................] duration:
675.470659ms
FAIL (2025-08-11 22:27:31 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 675.493002ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X...........X.....................................] duration:
564.714512ms
FAIL (2025-08-11 22:27:11 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 564.745342ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [...........................XX...............................] duration:
253.344042ms
FAIL (2025-08-11 22:26:55 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 253.373446ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X...........X....................................] duration:
1.741466478s
FAIL (2025-08-11 22:26:33 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.741491486s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [............................XX..............................] duration:
226.049763ms
FAIL (2025-08-11 22:26:05 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 226.076574ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X...........X...................................] duration:
668.232268ms
FAIL (2025-08-11 22:26:03 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 668.258888ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.............................XX.............................] duration:
218.904866ms
FAIL (2025-08-11 22:25:45 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 218.932728ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X...........X..................................] duration:
447.815696ms
FAIL (2025-08-11 22:25:05 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 447.840382ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.258408769s
FAIL (2025-08-11 22:24:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 16)
======LOGS======
0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment
Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments...
Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil>
Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments...
Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16)
Duration: 2.258437814s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............X...........X.................................] duration:
446.433312ms
FAIL (2025-08-11 22:24:49 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 446.455103ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [..............................XX............................] duration:
417.124122ms
FAIL (2025-08-11 22:24:49 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 417.146975ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
2.339739986s
FAIL (2025-08-11 22:23:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 16)
======LOGS======
0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment
Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments...
Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil>
Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments...
Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16)
Duration: 2.339761968s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [...............................XX...........................] duration:
377.115013ms
FAIL (2025-08-11 22:23:48 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 377.139508ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............X...........X................................] duration:
980.691634ms
FAIL (2025-08-11 22:23:37 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 980.719848ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [................................XX..........................] duration:
360.488001ms
FAIL (2025-08-11 22:23:02 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 360.504261ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [................X...........X...............................] duration:
834.475479ms
FAIL (2025-08-11 22:22:53 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 834.504214ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
3 out of 60 is failed (threshold=3) [XXX.........................................................] duration:
2.291322978s
FAIL (2025-08-11 22:22:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 16)
======LOGS======
0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment
Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments...
Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil>
Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments...
Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16)
Duration: 2.291352423s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X...........X..............................] duration:
843.200021ms
FAIL (2025-08-11 22:22:13 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 843.22129ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.................................XX.........................] duration:
555.966957ms
FAIL (2025-08-11 22:22:06 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 555.990051ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
3 out of 60 is failed (threshold=3) [.XXX........................................................] duration:
2.596037209s
FAIL (2025-08-11 22:21:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 16)
======LOGS======
0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment
Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments...
Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil>
Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments...
Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16)
Duration: 2.596081672s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [..................................XX........................] duration:
508.792504ms
FAIL (2025-08-11 22:21:41 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 508.822009ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................X...........X.............................] duration:
924.42016ms
FAIL (2025-08-11 22:21:25 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 924.451218ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration:
2.320577132s
FAIL (2025-08-11 22:21:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 16)
======LOGS======
0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment
Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments...
Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil>
Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments...
Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16)
Duration: 2.320599824s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [...................................XX.......................] duration:
364.067187ms
FAIL (2025-08-11 22:20:53 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 364.092114ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X...........X............................] duration:
1.079940092s
FAIL (2025-08-11 22:20:26 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.079956043s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
3 out of 60 is failed (threshold=3) [...XXX......................................................] duration:
2.395402591s
FAIL (2025-08-11 22:20:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 16)
======LOGS======
0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment
Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments...
Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil>
Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments...
Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16)
Duration: 2.395423301s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [....................................XX......................] duration:
414.579005ms
FAIL (2025-08-11 22:20:11 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 414.609253ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration:
2.422681437s
FAIL (2025-08-11 22:20:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 16)
======LOGS======
0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment
Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments...
Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil>
Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments...
Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16)
Duration: 2.422711473s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................X...........X...........................] duration:
600.713161ms
FAIL (2025-08-11 22:19:58 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 600.737429ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration:
446.946177ms
FAIL (2025-08-11 22:19:37 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 446.968099ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
3 out of 60 is failed (threshold=3) [.....XXX....................................................] duration:
2.305552083s
FAIL (2025-08-11 22:19:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 16)
======LOGS======
0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment
Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments...
Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil>
Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments...
Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16)
Duration: 2.30559237s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................X...........X..........................] duration:
1.076061864s
FAIL (2025-08-11 22:18:45 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.076097841s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
3 out of 60 is failed (threshold=3) [......XXX...................................................] duration:
2.398665428s
FAIL (2025-08-11 22:18:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 16)
======LOGS======
0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment
Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments...
Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil>
Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments...
Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16)
Duration: 2.398691147s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [......................................XX....................] duration:
184.71561ms
FAIL (2025-08-11 22:18:33 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 184.745387ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.......................................XX...................] duration:
213.310417ms
FAIL (2025-08-11 22:17:59 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 22:03:25.651495472 (76h37m6.382763121s) [upload=79689]: Starting upload Aug 11 22:03:25.651572135 (76h37m6.382839795s) [upload=79689]: Done starting upload Aug 11 22:03:25.651651094 (76h37m6.382918774s) [upload=79689]: Uploading segments... Aug 11 22:03:25.651710957 (76h37m6.382978637s) [upload=79689]: Got next segment. Inline: true Aug 11 22:03:25.651719252 (76h37m6.382986912s) [upload=79689]: Waiting for error group managing segments... Aug 11 22:03:25.651784644 (76h37m6.383052285s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 22:03:25.651793281 (76h37m6.383060961s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 11 22:03:25.651799393 (76h37m6.383067044s) [upload=79689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 11 22:03:25.654474785 (76h37m6.385742435s) [upload=79689]: Done uploading segments... Aug 11 22:03:25.654487048 (76h37m6.385754698s) [upload=79689]: Upload finished. err: metaclient: Too Many Requests
Duration: 213.335856ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
3 out of 60 is failed (threshold=3) [.......XXX..................................................] duration:
2.278405844s
FAIL (2025-08-11 22:17:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 16)
======LOGS======
0:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353442270 (76h15m5.0847099s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.353531519 (76h15m5.084799138s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Uploading piece...
Aug 11 21:41:24.805923381 (76h15m5.537191001s) [upload=79308]: Next returned nil segment
Aug 11 21:41:24.805937086 (76h15m5.537204707s) [upload=79308]: Waiting for error group managing segments...
Aug 11 21:41:24.967784283 (76h15m5.699051893s) [noise=true , address=89.149.201.147:4003 , piece_id=5JJVE7444RMELSKVU3EK6ZD2F3EZBMLVMTSJICUCIEA26RIBW3BA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.967963580 (76h15m5.6992312s) [noise=true , address=81.171.14.70:4048 , piece_id=RLNM63A27LKCIXCX5GFQ5AXWZYM7VQAPWXPNNIK47NXJUOKQTHPA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968225722 (76h15m5.699493343s) [noise=true , address=89.149.201.147:4002 , piece_id=NQW2FJWQGGDDVLB3RP2IFYG3UP74QDKJBXEK6LORS5JIH5SY4RAQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968545024 (76h15m5.699812654s) [noise=true , address=89.149.201.147:4005 , piece_id=NWVWPP34677QA2QQ6Q7L76EZOUHKFYGME6CZIUMZZO3UCUL5POWQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.968656423 (76h15m5.699924053s) [noise=true , address=89.149.201.147:4009 , piece_id=VIMVAPEQSMP3A4VQPN5LQRAPBQYD4MKXWBEMGQK3NID7224DZB6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.977823613 (76h15m5.709091234s) [noise=true , address=89.149.201.147:4024 , piece_id=KRLDIZGT5CN45WZW7M4HDKHRT6F4KTKEUS4WI4NC36CZGSAADGZA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979669626 (76h15m5.710937246s) [noise=true , address=81.171.14.70:4045 , piece_id=2AURLZFBBWM2VEY3JPOCZRUPIUBJX53AO4UOBIQAGOOERPOGICGA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.979993745 (76h15m5.711261375s) [noise=true , address=89.149.201.147:4020 , piece_id=WQLDFIAS2SVIYCYUAFVM6VAI5EOQ7F5BH5YBAGEPE4KMDVOJNIIQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.993790642 (76h15m5.725058263s) [noise=true , address=81.171.14.70:4025 , piece_id=IWREIHFBW3YADVG5OPHQBRVAMMBXJUBLT6UAO7LAKSNBBBXODL5Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:24.994609462 (76h15m5.725877082s) [noise=true , address=81.171.14.70:4037 , piece_id=E7FK5NJDE2ILFREUUYLG5UZSDNFJESOQEZVAXYXP2T6XNLLRESKA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000153229 (76h15m5.731420859s) [noise=true , address=89.149.201.147:4010 , piece_id=B5W5C53KPB2IJSHDA2UQQFGFCJ3D2XC6PP7BKOPKMNNMAK6BYDHA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.000273545 (76h15m5.731541175s) [noise=true , address=81.171.14.70:4044 , piece_id=EQEXHF67A23A2VJE7NUFS2A4XHNFBY5CRQ3LGWOAAVPYPOI7UTBA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.023125263 (76h15m5.754392883s) [noise=true , address=81.171.14.70:4036 , piece_id=OZB6PZT3E4SHFRY4MFJI6AFDAQTLINYXQCFCVUEHXGOCKE6LCVMQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.036563996 (76h15m5.767831626s) [noise=true , address=81.171.14.70:4039 , piece_id=5OZSB5VHVSIDE5N3P24WYMEM6WZ3PXZLORGBFTNF4HON6R5U3B4Q, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.037711014 (76h15m5.768978633s) [noise=true , address=81.171.14.70:4033 , piece_id=QKRLUCVQE524W4WWPWT5XUCKST3XMQVREVB3W32AJEJ7QQZWNFVA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:41:25.068155089 (76h15m5.799422709s) [noise=true , address=89.149.201.147:4022 , piece_id=3HN5PRZXPJZSY62SVUPMYOI7O6POKWMIDKGK53ONPSFCECEI2LXQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: <nil>
Aug 11 21:42:23.857391458 (76h16m4.588659088s) [noise=true , address=185.167.204.205:4022 , piece_id=FDVO4N5DOSBPL4JWJEACJDP3JZQ25GOAGXHXWTHVX26RUONNVK6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: rpc: dial tcp 185.167.204.205:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857430571 (76h16m4.588698201s) [noise=true , address=185.167.204.205:4016 , piece_id=KLUOWERRWDMBNHCSKW7LLJKB3GQMQ3MJQ3CA45BZ6IMD64JWZXJQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1scSMv7XDKsGKUMR8t1AdrQubP2142yorfVKaE4UyQoufi34TJ, address:185.167.204.205:4016): protocol: rpc: dial tcp 185.167.204.205:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857457712 (76h16m4.588725342s) [noise=true , address=185.167.204.206:4053 , piece_id=6SPASGLZOMLOX6YIJF6QI2VNNFCNULD32PPUSUOY7J6W6UMAU5NA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857468763 (76h16m4.588736393s) [noise=true , address=185.167.204.206:4064 , piece_id=2FMTSU6U7BSYK6V6SZ2TC72HHAIGB7E2IH6WKLD2JKLIZRDQ4M6A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857498157 (76h16m4.588765787s) [noise=true , address=185.167.204.206:4050 , piece_id=CRARH5UZZKK47D7QB5WMJBONSRPT4RKF5BPAUDNLHE67JXUJ56HQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857509490 (76h16m4.58877712s) [noise=true , address=185.167.204.205:4011 , piece_id=EWIX2W6CHL256HADY5JLN5YCFW3IMZZTOS2EYXQLRPMH6S35JFGQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857528346 (76h16m4.588795976s) [noise=true , address=185.167.204.205:4008 , piece_id=RWFQHSLWP54H6GYR6T3OCI44VNVTRQTPVVBJ3IARTBTH2WMVYTUA, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857556748 (76h16m4.588824369s) [noise=true , address=185.167.204.205:4014 , piece_id=VTQP7QOZMCJR6R73KENVBHXJ7PWD5MP3X2RWWUAXQQDZ4SR5LT4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857561528 (76h16m4.588829178s) [noise=true , address=185.167.204.205:4001 , piece_id=5FGTFLPBMN72AJDTMWKATCQV46BUF5H7G36IFHFUGN7DGEBXYBFQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12gZXkE3Cnmu2oyKMmUfaADZ4LDPQahTcE9sUwW173on719Bb91, address:185.167.204.205:4001): protocol: rpc: dial tcp 185.167.204.205:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857584801 (76h16m4.588852431s) [noise=true , address=185.167.204.206:4047 , piece_id=7VQ4OY3QL3YMXVW5TLZ5B5LJVGQYTJWQXRAAIL5Q47RLRXDD4KQQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: rpc: dial tcp 185.167.204.206:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857610229 (76h16m4.588877849s) [noise=true , address=185.167.204.205:4028 , piece_id=A7XCV5SIGJ343GRER376LPMWJVCSCR2UKZ2HCSYDCWFHTOWEB3ZQ, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1HZoGwBuBh9Kg7gkvH9pfrjtxQurwxdnF7pWfycGuLHrHTSq61, address:185.167.204.205:4028): protocol: rpc: dial tcp 185.167.204.205:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857615549 (76h16m4.588883169s) [noise=true , address=185.167.204.206:4038 , piece_id=NIDYWBI5NTGLMTA5RGIMQ6WXPQBGZA3OBX6KQNAUM2G2Y4GADG4A, seg_pos={0 0}, upload=79308]: Done uploading piece. err: ecclient: upload failed (node:1jtjiXHQRFhyw6vb8A1RDPfe4j2283znmAZiZCBmrvFpQrqJiT, address:185.167.204.206:4038): protocol: write tcp 5.161.89.252:62812->185.167.204.206:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 11 21:42:23.857628543 (76h16m4.588896173s) [seg_pos={0 0}, upload=79308]: Done waiting for segment. successful: 16 optimal: 26 errs: <nil>
Aug 11 21:42:23.969512092 (76h16m4.700779722s) [upload=79308]: Done uploading segments...
Aug 11 21:42:23.969540085 (76h16m4.700807715s) [upload=79308]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 16)
Duration: 2.27842979s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................X...........X.........................] duration:
1.282799202s
FAIL (2025-08-11 22:17:41 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 21:58:40.674996400 (76h32m21.40626403s) [upload=79603]: Starting upload Aug 11 21:58:40.675045133 (76h32m21.406312763s) [upload=79603]: Done starting upload Aug 11 21:58:40.675097099 (76h32m21.406364729s) [upload=79603]: Uploading segments... Aug 11 21:58:40.675263623 (76h32m21.406531253s) [upload=79603]: Got next segment. Inline: false Aug 11 21:58:40.675310160 (76h32m21.40657779s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 11 21:58:40.675314719 (76h32m21.406582339s) [upload=79603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 11 21:58:40.678204004 (76h32m21.409471634s) [upload=79603]: Next segment err: metaclient: Too Many Requests Aug 11 21:58:40.678247115 (76h32m21.409514745s) [upload=79603]: Done uploading segments... Aug 11 21:58:40.678256512 (76h32m21.409524132s) [upload=79603]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.282849187s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|