Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 121526 reports (Page 1008 of 2431)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.838233497s FAIL (2025-09-04 11:02:24 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.838263765s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.909356673s FAIL (2025-09-04 11:01:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.909367503s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000372326s FAIL (2025-09-04 11:01:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000387805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 21.642110493s FAIL (2025-09-04 11:01:37 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 21.642126113s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000458832s FAIL (2025-09-04 11:01:15 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000477408s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 467.960818ms FAIL (2025-09-03 21:34:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 467.990916ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 328.153889ms FAIL (2025-09-03 21:34:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 328.175198ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 579.082371ms FAIL (2025-09-03 21:34:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 579.09321ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.XXX........................................................] duration: 721.614033ms FAIL (2025-09-03 21:33:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 721.632617ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration: 485.9861ms FAIL (2025-09-03 21:32:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 486.038177ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...XXX......................................................] duration: 606.124977ms FAIL (2025-09-03 21:30:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 606.146547ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration: 603.84526ms FAIL (2025-09-03 21:29:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 603.863604ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....XXX....................................................] duration: 393.037528ms FAIL (2025-09-03 21:28:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 393.062124ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......XXX...................................................] duration: 833.456549ms FAIL (2025-09-03 21:28:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 833.473191ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......XXX..................................................] duration: 988.864964ms FAIL (2025-09-03 21:27:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 988.882548ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........XXX.................................................] duration: 1.057146393s FAIL (2025-09-03 21:26:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.057167442s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........XXX................................................] duration: 758.698213ms FAIL (2025-09-03 21:26:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 758.736083ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 413.139805ms FAIL (2025-09-03 21:26:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 413.159872ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 363.195576ms FAIL (2025-09-03 21:25:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 363.214422ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........XXX...............................................] duration: 582.070338ms FAIL (2025-09-03 21:25:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 582.092489ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 460.295603ms FAIL (2025-09-03 21:24:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 460.319459ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.X........................................................] duration: 435.67509ms FAIL (2025-09-03 21:24:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 435.697862ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........XXX..............................................] duration: 844.640529ms FAIL (2025-09-03 21:23:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 844.659705ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............XXX.............................................] duration: 431.008998ms FAIL (2025-09-03 21:23:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 431.069482ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.X.X.......................................................] duration: 422.768363ms FAIL (2025-09-03 21:23:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 422.834026ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............XXX............................................] duration: 385.493145ms FAIL (2025-09-03 21:22:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 385.513624ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX.X.X......................................................] duration: 755.27022ms FAIL (2025-09-03 21:22:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 755.301348ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............XXX...........................................] duration: 485.460881ms FAIL (2025-09-03 21:21:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 485.703588ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXX.X.X.....................................................] duration: 366.551703ms FAIL (2025-09-03 21:21:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 366.573614ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXX.X.X....................................................] duration: 1.295653458s FAIL (2025-09-03 21:21:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.295668897s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............XXX..........................................] duration: 414.574195ms FAIL (2025-09-03 21:20:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 414.602939ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XXXX.X.X...................................................] duration: 432.233962ms FAIL (2025-09-03 21:20:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 432.263307ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................XXX.........................................] duration: 363.259644ms FAIL (2025-09-03 21:19:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 363.278198ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XXXX.X.X..................................................] duration: 385.726153ms FAIL (2025-09-03 21:19:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 385.759767ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XXXX.X.X.................................................] duration: 394.540757ms FAIL (2025-09-03 21:18:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 394.566476ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................XXX........................................] duration: 381.907815ms FAIL (2025-09-03 21:18:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 381.963489ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................XXX.......................................] duration: 473.261489ms FAIL (2025-09-03 21:17:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 473.281156ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XXXX.X.X................................................] duration: 801.07059ms FAIL (2025-09-03 21:17:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 801.103141ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................XXX......................................] duration: 717.409116ms FAIL (2025-09-03 21:16:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 717.434034ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [X....XXXX.X.X...............................................] duration: 596.591261ms FAIL (2025-09-03 21:16:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 596.627228ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................XXX.....................................] duration: 451.349273ms FAIL (2025-09-03 21:16:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 451.366555ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X....XXXX.X.X..............................................] duration: 855.518383ms FAIL (2025-09-03 21:15:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 855.55876ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................XXX....................................] duration: 605.945068ms FAIL (2025-09-03 21:15:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 605.970306ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X....XXXX.X.X.............................................] duration: 416.167842ms FAIL (2025-09-03 21:15:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 416.194172ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................XXX...................................] duration: 487.34944ms FAIL (2025-09-03 21:14:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.372504ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X....XXXX.X.X............................................] duration: 531.549137ms FAIL (2025-09-03 21:13:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 531.568473ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X....XXXX.X.X...........................................] duration: 421.606839ms FAIL (2025-09-03 21:13:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 421.631997ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................XXX..................................] duration: 410.994805ms FAIL (2025-09-03 21:12:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 411.006577ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................XXX.................................] duration: 754.96011ms FAIL (2025-09-03 21:12:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 754.98135ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X....XXXX.X.X..........................................] duration: 445.986568ms FAIL (2025-09-03 21:12:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 446.010163ms core uplink us-select-1 us1-placement-20

1
...
1006
1007
1008
1009
1010
...
2431