Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 371.334682ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:26:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 371.378293ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 343.446355ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:25:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 343.46433ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 398.631372ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:25:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 398.649016ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 340.687087ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:24:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 340.700884ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 374.315319ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:24:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 374.334796ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 389.136194ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:23:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 389.155791ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 358.122962ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:22:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 358.145164ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 357.609071ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:22:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 357.627264ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 429.211618ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:21:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 429.252144ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 374.241692ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:20:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 374.261609ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 336.194506ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:20:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 336.213541ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 348.267202ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:19:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 348.288072ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 345.139917ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:19:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 345.160886ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 370.661483ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:18:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 370.679396ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 381.004534ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:17:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 381.021236ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 382.532288ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:16:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 382.552307ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 374.04964ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:15:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 374.066293ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 369.465058ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:15:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 369.482069ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 363.222125ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:14:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 363.239268ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 388.666007ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:14:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 388.682077ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 366.627618ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:13:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 366.644741ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 460.9314ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:13:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 460.948352ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 356.439698ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:12:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 356.479864ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 380.764847ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:11:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 380.776698ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 361.559328ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:10:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 361.580397ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 394.725657ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:09:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 394.746036ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 375.16688ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:08:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 375.18249ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 360.618486ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:07:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 360.633434ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 418.150128ms trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:07:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 418.165346ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001596899s trace: https://jaeger.dev.storj.io/trace/5859ea4e67fe575a FAIL (2025-12-07 16:07:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001627516s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 449.402186ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:48:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 449.422716ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 425.255922ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:48:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 425.268385ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 441.110888ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:47:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.125045ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 455.792382ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:46:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.808271ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 431.547814ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:46:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 431.572551ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 476.721068ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:45:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 476.741074ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 354.315665ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:45:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 354.32895ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 488.529135ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:44:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 488.546838ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 444.261463ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:43:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.28196ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 360.843109ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:42:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 360.861383ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 428.84433ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:41:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 428.870889ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 443.178265ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:40:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 443.198983ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 470.700277ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:39:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.721246ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 550.159806ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:38:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 550.179413ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 558.972718ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:37:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 558.996412ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 431.365119ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:37:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 431.385766ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 416.042156ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:36:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 416.179673ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 452.595758ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:36:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 452.606006ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 419.88228ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:35:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 419.943775ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 465.778353ms trace: https://jaeger.dev.storj.io/trace/5ee917a62999ce3b FAIL (2025-12-07 14:34:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 14:05:12.793746279 (46h36m0.491314615s) [upload=40891]: Starting upload Dec 7 14:05:12.793820377 (46h36m0.491388713s) [upload=40891]: Done starting upload Dec 7 14:05:12.794040619 (46h36m0.491608935s) [upload=40891]: Uploading segments... Dec 7 14:05:12.794207350 (46h36m0.491775685s) [upload=40891]: Got next segment. Inline: false Dec 7 14:05:12.794270879 (46h36m0.491839214s) [upload=40891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 14:05:12.794277000 (46h36m0.491845346s) [upload=40891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 14:05:12.815883299 (46h36m0.513451636s) [seg_pos={0 0}, upload=40891]: Begin upload segment... Dec 7 14:05:12.815956126 (46h36m0.513524472s) [seg_pos={0 0}, upload=40891]: Done begin upload segment. Dec 7 14:05:12.815973168 (46h36m0.513541474s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.815992825 (46h36m0.513561161s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.816593907 (46h36m0.514162243s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817033549 (46h36m0.514601884s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.817608943 (46h36m0.515177279s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818051330 (46h36m0.515619666s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.818512791 (46h36m0.516081127s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.819185398 (46h36m0.516753734s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.821078003 (46h36m0.518646339s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.822854892 (46h36m0.520423238s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.824567812 (46h36m0.522136148s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.826399914 (46h36m0.52396823s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.827587042 (46h36m0.525155388s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.828083339 (46h36m0.525651645s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.829168395 (46h36m0.526736702s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.830278419 (46h36m0.527846735s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Uploading piece... Dec 7 14:05:12.876303152 (46h36m0.573871468s) [noise=true , address=77.90.31.16:4734 , piece_id=EU5C6XZ3NAU7AXJ6T2X3YNE5TLN44UD5OTKHWDW7AMMH2OWUB7QQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.877232387 (46h36m0.574800703s) [noise=true , address=145.79.227.30:4763 , piece_id=TMPQE3SUM2SWP2RO3YGACVUIAVHWV2BUUX522XCSJBACMEQJT2IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 7 14:05:12.916831066 (46h36m0.614399382s) [upload=40891]: Next returned nil segment Dec 7 14:05:12.916839262 (46h36m0.614407567s) [upload=40891]: Waiting for error group managing segments... Dec 7 14:05:12.929551988 (46h36m0.627120294s) [noise=true , address=145.79.227.42:4123 , piece_id=ISX3YLJJQ6L4HMTGY2J47EZM74UZJA5M65JN5TOFY6IAA2DHPABQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.929866526 (46h36m0.627434872s) [noise=true , address=145.79.227.27:4689 , piece_id=YDNSB4BUQDP7CNIAU73X6ILCKRWUYQWQEIUA2IYFPLPDGNNJ56YQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930150587 (46h36m0.627718893s) [noise=true , address=145.79.227.43:4159 , piece_id=R7UD47XJ7XGLARQAFNL2ISONIR6VZLFJ3PWWQ5V4ZKYB7KMD6MCA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930672422 (46h36m0.628240757s) [noise=true , address=145.79.227.29:4941 , piece_id=ROH6OOR3PU6F6YGZMGBDVL7IZI2YFW6WOMALTBPE2DKDBH35D43Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.930722245 (46h36m0.628290561s) [noise=true , address=145.79.227.41:4086 , piece_id=35MLDKAKB7U26I33RKISI33JYRZK36NVREY4H43LQ2GJ7IEEW4EA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931139735 (46h36m0.62870804s) [noise=true , address=145.79.227.32:4834 , piece_id=PD7R4WLJLLZBCFND6BUPEJ3LIJBKNETFMLB2C2X73HH3Q4APZ6KQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931585557 (46h36m0.629153894s) [noise=true , address=145.79.227.38:4977 , piece_id=3EAL7Y6XNXCRCNLW2SX7EADFYVVZIWQ4AJ5BQMIQV3KJKCC4C4IQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.931807582 (46h36m0.629375888s) [noise=true , address=145.79.227.23:4546 , piece_id=FTUSB2QRIKNNSIN67M5EZYRSM6MUYMC746TLBLZXTYEXLVE7MYTQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932398866 (46h36m0.629967213s) [noise=true , address=145.79.227.40:4051 , piece_id=NS2P7FBECXVP2LCCF6UQ4BDDL7IXN4BWLRTIT6MAUOF2C5FVFI3A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.932913437 (46h36m0.630481743s) [noise=true , address=145.79.227.43:4158 , piece_id=C7JLSPLTCF7SJVKHDYAOTLARPNSDHNU3ARV2FJQMEUR4QZOWOH3Q, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.936343114 (46h36m0.63391143s) [noise=true , address=145.79.227.31:4797 , piece_id=6NHGY624TXJ2NL4KGLCKR5TETBSH6R4KF23O3RG3LYR7OOKJ55ZA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:12.938164697 (46h36m0.635733033s) [noise=true , address=45.91.116.11:4117 , piece_id=WELLJ6VZNKBHBNFTK7SZZ72J2EKFBVA6SCB27HX7JIMQUYXPY5BA, seg_pos={0 0}, upload=40891]: Done uploading piece. err: <nil> Dec 7 14:05:42.780287323 (46h36m30.477855639s) [noise=true , address=77.90.31.17:4772 , piece_id=5C5RYNJHJOR74O6MNAGKIAHGBZLGT7ZYAPHG4CJI655GS2UFFGCQ, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780406927 (46h36m30.477975233s) [noise=true , address=77.90.31.12:4592 , piece_id=WHMBDMM2OKNWDRBIUYQG4N4S4BIHPSA3C2TNAEJ7BGRMDK5G726A, seg_pos={0 0}, upload=40891]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 14:05:42.780455096 (46h36m30.478023402s) [seg_pos={0 0}, upload=40891]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 14:05:42.813667787 (46h36m30.511236093s) [upload=40891]: Done uploading segments... Dec 7 14:05:42.813692484 (46h36m30.5112608s) [upload=40891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 465.80333ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)