Orbiter History

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 779.972594ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:16:09 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 779.995586ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 842.880569ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:15:49 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 842.91846ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 279.966488ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:15:32 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 279.99012ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 285.286525ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:14:28 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 285.306753ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 283.099129ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:13:43 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 283.122963ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 297.238624ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:13:02 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 297.249814ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 287.90659ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:12:11 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 287.961071ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 310.886138ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:11:47 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 310.915814ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 278.177275ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:11:12 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 278.193246ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 290.345478ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:09:59 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 290.366567ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 275.741695ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:09:38 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 275.76023ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 288.381407ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:09:05 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 288.401744ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 283.781824ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:08:19 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 283.803174ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 292.592216ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:07:58 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 292.609708ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 286.339321ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:07:15 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 286.361443ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 840.144194ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:06:07 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 840.162728ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 275.24082ms remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:05:09 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 275.256729ms edge linksharing bgp ap1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 2m0.000582445s remote: "tcp:136.0.77.4:443" target: https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/354840a666ad7466 FAIL (2025-12-21 01:05:08 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 2m0.000602281s edge linksharing bgp ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 455.844074ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:13:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 455.865232ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 325.758061ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:12:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 325.776195ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 359.887747ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:11:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 359.903917ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 310.677087ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:10:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 310.69932ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 342.776843ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:09:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 342.812418ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 402.350162ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:09:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 402.367875ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 400.926945ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:08:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 400.971577ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 329.110404ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:07:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 329.131623ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 364.241672ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:07:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 364.26739ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 366.355721ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:06:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 366.373144ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 398.014795ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:06:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 398.035203ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 446.077311ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:05:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 446.098519ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 359.470735ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:05:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 359.494208ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 438.600632ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:04:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 438.621591ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 379.817069ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:03:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 379.838801ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 374.591988ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:02:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 374.613208ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 411.841335ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:02:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 411.861201ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 336.986117ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:01:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 337.001926ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 375.746183ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-21 00:00:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 375.773264ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 368.032524ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:59:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 368.058103ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 457.46394ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:58:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 457.491832ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 360.698362ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:58:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 360.727207ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 484.299282ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:57:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 484.322695ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 330.087556ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:57:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 330.115087ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 469.456084ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:56:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 469.478363ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 324.613583ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:56:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 324.636577ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 315.53817ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:55:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 315.551635ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 375.170318ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:55:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 375.186618ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 410.791583ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:54:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 410.814315ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 337.536212ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:53:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 337.551861ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 387.096296ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:52:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 387.120731ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 370.543034ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:51:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 370.572709ms core uplink us-select-1

Orbiter Check History

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)