Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 544.026142ms trace: https://jaeger.dev.storj.io/trace/2dad800a9f813073 FAIL (2025-12-11 02:15:43 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 544.055636ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 578.778057ms trace: https://jaeger.dev.storj.io/trace/59e683f9bb1a8a96 FAIL (2025-12-11 02:15:20 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 578.797354ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 597.237982ms trace: https://jaeger.dev.storj.io/trace/59e683f9bb1a8a96 FAIL (2025-12-11 02:14:57 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 597.258349ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 498.821998ms trace: https://jaeger.dev.storj.io/trace/2dad800a9f813073 FAIL (2025-12-11 02:14:41 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 498.915612ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 620.750436ms trace: https://jaeger.dev.storj.io/trace/59e683f9bb1a8a96 FAIL (2025-12-11 02:14:10 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 620.789129ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 532.413652ms trace: https://jaeger.dev.storj.io/trace/2dad800a9f813073 FAIL (2025-12-11 02:13:27 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 532.443698ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 651.545908ms trace: https://jaeger.dev.storj.io/trace/59e683f9bb1a8a96 FAIL (2025-12-11 02:13:01 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 651.570543ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 489.746973ms trace: https://jaeger.dev.storj.io/trace/2dad800a9f813073 FAIL (2025-12-11 02:12:56 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 489.760447ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 620.963522ms trace: https://jaeger.dev.storj.io/trace/59e683f9bb1a8a96 FAIL (2025-12-11 02:12:39 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 621.103474ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 643.03622ms trace: https://jaeger.dev.storj.io/trace/59e683f9bb1a8a96 FAIL (2025-12-11 02:12:21 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 643.0519ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 649.136135ms trace: https://jaeger.dev.storj.io/trace/2dad800a9f813073 FAIL (2025-12-11 02:11:45 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 649.187501ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 599.44225ms trace: https://jaeger.dev.storj.io/trace/59e683f9bb1a8a96 FAIL (2025-12-11 02:11:39 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 599.461105ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 676.002757ms trace: https://jaeger.dev.storj.io/trace/2dad800a9f813073 FAIL (2025-12-11 02:10:56 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 676.019819ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 574.153113ms trace: https://jaeger.dev.storj.io/trace/59e683f9bb1a8a96 FAIL (2025-12-11 02:10:52 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 574.17245ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 580.337838ms trace: https://jaeger.dev.storj.io/trace/59e683f9bb1a8a96 FAIL (2025-12-11 02:10:23 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 580.356063ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 452.406765ms trace: https://jaeger.dev.storj.io/trace/2dad800a9f813073 FAIL (2025-12-11 02:10:10 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 452.417986ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 23.208869ms trace: https://jaeger.dev.storj.io/trace/59e683f9bb1a8a96 FAIL (2025-12-11 02:09:32 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 23.224577ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 22.463706ms trace: https://jaeger.dev.storj.io/trace/2dad800a9f813073 FAIL (2025-12-11 02:09:05 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 22.480026ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 24.386758ms trace: https://jaeger.dev.storj.io/trace/65cb94e8e1f630b6 FAIL (2025-12-11 02:09:05 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 24.399061ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 593.130647ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:31:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 593.154882ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 437.081493ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:30:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 437.10097ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 457.394501ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:30:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 457.411972ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 550.039509ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:29:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 550.058103ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 402.698614ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:29:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 402.718461ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXXXX......................................................] duration: 474.447988ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:28:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 474.490396ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXXXXXX.....................................................] duration: 585.506735ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:27:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 585.523988ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 858.703299ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:26:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 858.721663ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXXXXXXX....................................................] duration: 428.598881ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:26:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 428.61527ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXXXXXXXX...................................................] duration: 428.713866ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:26:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 428.731279ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXXXXXXXXX..................................................] duration: 431.386276ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:26:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 431.402667ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 427.206838ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:25:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.221417ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 460.78163ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:25:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 460.802148ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXXXXXXXXX.................................................] duration: 808.313109ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:24:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 808.330891ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 452.054185ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:24:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 452.071477ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 414.09337ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:23:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 414.11432ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXXXXXXXXXX................................................] duration: 462.809906ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:23:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 462.823903ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 467.670687ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:23:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.6882ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX...............................................] duration: 444.167442ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:22:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 444.180897ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 414.220628ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:22:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 414.289227ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 456.3664ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:21:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.386558ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX..............................................] duration: 391.362518ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:21:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 391.385541ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 524.82179ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:20:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 524.839775ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.............................................] duration: 648.323913ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:20:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 648.345063ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 405.531878ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:20:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 405.551226ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 459.362236ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:19:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 459.382274ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX............................................] duration: 383.952304ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:19:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 383.973774ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 436.606617ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:18:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 436.619792ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX...........................................] duration: 510.896757ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:18:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 510.91446ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX..........................................] duration: 520.029557ms trace: https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3 FAIL (2025-12-11 00:17:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece... Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments... Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil> Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments... Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 520.063551ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 423.029123ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-11 00:17:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 423.046956ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)