Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............................X....XXXX.X.X.................] duration: 456.522132ms FAIL (2025-09-03 20:54:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 456.56867ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.X.X.X.......XX.....X...........................XXX......] duration: 767.224581ms FAIL (2025-09-03 20:53:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 767.24496ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................X....XXXX.X.X................] duration: 910.536694ms FAIL (2025-09-03 20:53:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 910.557983ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [................................X....XXXX.X.X...............] duration: 747.682911ms FAIL (2025-09-03 20:52:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 747.714118ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X.X.X.X.......XX.....X...........................XXX.....] duration: 834.308543ms FAIL (2025-09-03 20:52:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 834.329251ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X...X.X.X.X.......XX.....X...........................XXX....] duration: 570.825892ms FAIL (2025-09-03 20:52:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 570.876828ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................................X....XXXX.X.X..............] duration: 651.613725ms FAIL (2025-09-03 20:51:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 651.640044ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X...X.X.X.X.......XX.....X...........................XXX...] duration: 410.836634ms FAIL (2025-09-03 20:51:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 410.856873ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................................X....XXXX.X.X.............] duration: 800.495489ms FAIL (2025-09-03 20:50:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 800.543259ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X...X.X.X.X.......XX.....X...........................XXX..] duration: 459.134603ms FAIL (2025-09-03 20:50:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 459.173415ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................................X....XXXX.X.X............] duration: 976.606949ms FAIL (2025-09-03 20:50:22 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 976.637326ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X..X...X.X.X.X.......XX.....X...........................XXX.] duration: 777.466209ms FAIL (2025-09-03 20:50:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 777.484002ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX..X...X.X.X.X.......XX.....X...........................XXX] duration: 30.000983806s FAIL (2025-09-03 20:49:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000999155s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................................X....XXXX.X.X...........] duration: 1.353606237s FAIL (2025-09-03 20:49:27 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.353633777s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX..X...X.X.X.X.......XX.....X...........................XX] duration: 30.000528605s FAIL (2025-09-03 20:48:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000557309s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................................X....XXXX.X.X..........] duration: 840.999252ms FAIL (2025-09-03 20:48:30 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 841.021183ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.XX..X...X.X.X.X.......XX.....X...........................X] duration: 30.000473314s FAIL (2025-09-03 20:47:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000494695s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................................X....XXXX.X.X.........] duration: 833.932447ms FAIL (2025-09-03 20:47:26 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 833.954799ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX.XX..X...X.X.X.X.......XX.....X...........................] duration: 457.295624ms FAIL (2025-09-03 20:46:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 457.313988ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................................X....XXXX.X.X........] duration: 840.116612ms FAIL (2025-09-03 20:46:26 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 840.135558ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX.XX..X...X.X.X.X.......XX.....X..........................] duration: 539.330388ms FAIL (2025-09-03 20:45:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 539.351928ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.XX.XX..X...X.X.X.X.......XX.....X.........................] duration: 445.9905ms FAIL (2025-09-03 20:45:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 446.010658ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................................X....XXXX.X.X.......] duration: 880.459325ms FAIL (2025-09-03 20:45:18 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 880.477338ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX.XX.XX..X...X.X.X.X.......XX.....X........................] duration: 400.825384ms FAIL (2025-09-03 20:44:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 400.852904ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................................X....XXXX.X.X......] duration: 13.718674467s FAIL (2025-09-03 20:44:39 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.718721134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XX.XX.XX..X...X.X.X.X.......XX.....X.......................] duration: 445.829628ms FAIL (2025-09-03 20:44:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 445.856169ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 415.590928ms FAIL (2025-09-03 20:43:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 415.617599ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..XX.XX.XX..X...X.X.X.X.......XX.....X......................] duration: 680.239718ms FAIL (2025-09-03 20:43:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 680.258493ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................................X....XXXX.X.X.....] duration: 673.585029ms FAIL (2025-09-03 20:43:11 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 673.609995ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...XX.XX.XX..X...X.X.X.X.......XX.....X.....................] duration: 530.360033ms FAIL (2025-09-03 20:42:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 530.373057ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 356.001117ms FAIL (2025-09-03 20:42:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 356.017447ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................................X....XXXX.X.X....] duration: 977.547967ms FAIL (2025-09-03 20:42:27 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 977.576479ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 586.067649ms FAIL (2025-09-03 20:42:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 586.096954ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................................X....XXXX.X.X...] duration: 1.123578013s FAIL (2025-09-03 20:42:07 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.123595816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..X........................................................] duration: 458.291916ms FAIL (2025-09-03 20:42:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 458.31604ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X...XX.XX.XX..X...X.X.X.X.......XX.....X....................] duration: 456.292857ms FAIL (2025-09-03 20:41:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.314839ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............................................X....XXXX.X.X..] duration: 1.087557396s FAIL (2025-09-03 20:41:32 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.087578805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration: 551.484702ms FAIL (2025-09-03 20:41:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 551.533593ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............................................X....XXXX.X.X.] duration: 856.378992ms FAIL (2025-09-03 20:41:02 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 856.397556ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................................X....XXXX.X.X] duration: 1m0.019737719s FAIL (2025-09-03 20:41:01 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.019833288s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX...XX.XX.XX..X...X.X.X.X.......XX.....X...................] duration: 446.846943ms FAIL (2025-09-03 20:40:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 446.867804ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.X..X......................................................] duration: 639.459492ms FAIL (2025-09-03 20:40:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 639.47988ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXX...XX.XX.XX..X...X.X.X.X.......XX.....X..................] duration: 482.844916ms FAIL (2025-09-03 20:40:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 482.867591ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................X....XXXX.X.] duration: 1.132626986s FAIL (2025-09-03 20:39:47 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== SegmentBegin Sep 3 20:38:45.917755653 (126h30m21.270540761s) [seg_pos={0 0}, upload=111540]: Begin upload segment... Sep 3 20:38:45.917803734 (126h30m21.270588832s) [seg_pos={0 0}, upload=111540]: Done begin upload segment. Sep 3 20:38:45.917825805 (126h30m21.270610903s) [noise=true , address=104.36.82.55:4223 , piece_id=75KSFJRWPIELGCS7HESLVYOGDWSB3DNFLNJVMGE45F4ZZCZXF2XQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.917835584 (126h30m21.270620681s) [noise=true , address=104.36.82.4:4818 , piece_id=X5PGQAM3LWG54G2XXY4XYDZV5UYON636CHPDFGMZRQPP4A7SWPWA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.917907188 (126h30m21.270692287s) [noise=true , address=104.36.82.62:4436 , piece_id=3IXS5X7WSNNVJ7APQOOIOL4YCQB7S4C5BJEVWWNOZJGVQIYKA65A, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918164122 (126h30m21.27094923s) [noise=true , address=79.127.205.245:4227 , piece_id=WTVDUD4FUM5F2NGVCOM2DGJSB7JCFOAO66GJVFD6DUA6U27MRPTQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918177106 (126h30m21.270962204s) [noise=true , address=104.36.82.20:4458 , piece_id=PXU2IWGZD3MNTSSZ6EUCQIEPYYAKCEUJDCLV6WM6SPRLASTLHUOA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918463093 (126h30m21.271248201s) [noise=true , address=77.90.31.13:4640 , piece_id=3IODZBJFSVVGFAJQZWDRXGZZSSD6TZKOPZ5UX2JBIEINN45R2INA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918558682 (126h30m21.271343791s) [noise=true , address=145.79.227.2:4118 , piece_id=UCZIMO7GNDYOR4VBQTK5G4O7I72MFGF6VGX2TXVSLDCCRNHBMNKQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918709266 (126h30m21.271494364s) [noise=true , address=104.36.82.44:4893 , piece_id=ACUTIIWFPHOWUNP6FNAWNKSK62WAZV6HXZC3SC645S4WJB5KKAUA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918930031 (126h30m21.271715129s) [noise=true , address=104.36.82.61:4388 , piece_id=UKZLF3QHDC7PV3QN3DNWZA6ZNX5LOJJTAE2ZTJWHSMLKQ6XO6FSA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918979534 (126h30m21.271764642s) [noise=true , address=104.36.82.56:4229 , piece_id=U5RTJWHM2PCHGSCGUU5XYFP6CZWC7J7VP6INSZXWKXIX5RBFSMIA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.919267074 (126h30m21.272052183s) [noise=true , address=104.36.82.60:4364 , piece_id=F6XATTDQKD6UNPZ36IHNV6BNP2FCZJOMGOVPE7HJHO5DM7W6ARZQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.919536050 (126h30m21.272321158s) [noise=true , address=104.36.82.10:4103 , piece_id=NR2RNEMTPDTOKRDFBKSEXT4IDC2HBNO5NIBVKMCYCRXXOMAABM3Q, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.919611963 (126h30m21.272397071s) [noise=true , address=45.91.116.7:4013 , piece_id=BMWM5FQ2VADDLBLMRU3DPC2CDEJCPIOAMXP7XCQFVGHCBJDUFIXA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.919772324 (126h30m21.272557422s) [noise=true , address=77.90.31.14:4692 , piece_id=3ZW7EUFU37VUYGXLT47MXKX6V5JHINNW4CVUTRSTIQBUPHCZWZTQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.919994261 (126h30m21.272779369s) [noise=true , address=104.36.82.23:4465 , piece_id=V7RWRCEEU5MCUZH26DZ3F6S7EDWDQ7QAMVNLY52J3KAMRGBOPP5A, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.920030039 (126h30m21.272815147s) [noise=true , address=77.90.31.16:4763 , piece_id=4AXJUIVFMYTFZT623PFX66JB2C4IT253XBW73F2R2EJY4KFPAODA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.920258829 (126h30m21.273043937s) [noise=true , address=104.36.82.12:4164 , piece_id=D3B3V2LFGLMKJH35G2YR3LLW6HJGVIWPDW2L4I2M6OAYFFDPAOBQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.920499361 (126h30m21.27328447s) [noise=true , address=64.49.13.27:4961 , piece_id=IGQXLHTJ22IIDFMFQTFVHO6WLQVF2H2OZ2N2QHGYFJ3URJBW3AOQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.994181494 (126h30m21.346966591s) [upload=111540]: Next returned nil segment Sep 3 20:38:45.994202032 (126h30m21.34698714s) [upload=111540]: Waiting for error group managing segments... Sep 3 20:38:46.014168995 (126h30m21.366954103s) [noise=true , address=104.36.82.62:4436 , piece_id=3IXS5X7WSNNVJ7APQOOIOL4YCQB7S4C5BJEVWWNOZJGVQIYKA65A, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.017977745 (126h30m21.370762842s) [noise=true , address=77.90.31.16:4763 , piece_id=4AXJUIVFMYTFZT623PFX66JB2C4IT253XBW73F2R2EJY4KFPAODA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.020311932 (126h30m21.37309704s) [noise=true , address=104.36.82.61:4388 , piece_id=UKZLF3QHDC7PV3QN3DNWZA6ZNX5LOJJTAE2ZTJWHSMLKQ6XO6FSA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.028096806 (126h30m21.380881914s) [noise=true , address=104.36.82.60:4364 , piece_id=F6XATTDQKD6UNPZ36IHNV6BNP2FCZJOMGOVPE7HJHO5DM7W6ARZQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.030036893 (126h30m21.38282199s) [noise=true , address=145.79.227.2:4118 , piece_id=UCZIMO7GNDYOR4VBQTK5G4O7I72MFGF6VGX2TXVSLDCCRNHBMNKQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.034361702 (126h30m21.387146799s) [noise=true , address=45.91.116.7:4013 , piece_id=BMWM5FQ2VADDLBLMRU3DPC2CDEJCPIOAMXP7XCQFVGHCBJDUFIXA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.042340099 (126h30m21.395125207s) [noise=true , address=77.90.31.13:4640 , piece_id=3IODZBJFSVVGFAJQZWDRXGZZSSD6TZKOPZ5UX2JBIEINN45R2INA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.047800253 (126h30m21.400585351s) [noise=true , address=104.36.82.23:4465 , piece_id=V7RWRCEEU5MCUZH26DZ3F6S7EDWDQ7QAMVNLY52J3KAMRGBOPP5A, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.048716665 (126h30m21.401501763s) [noise=true , address=77.90.31.14:4692 , piece_id=3ZW7EUFU37VUYGXLT47MXKX6V5JHINNW4CVUTRSTIQBUPHCZWZTQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.126456876 (126h30m21.479241974s) [noise=true , address=104.36.82.12:4164 , piece_id=D3B3V2LFGLMKJH35G2YR3LLW6HJGVIWPDW2L4I2M6OAYFFDPAOBQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.446267163 (126h30m21.799052291s) [noise=true , address=79.127.205.245:4227 , piece_id=WTVDUD4FUM5F2NGVCOM2DGJSB7JCFOAO66GJVFD6DUA6U27MRPTQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.539567192 (126h30m21.8923523s) [noise=true , address=64.49.13.27:4961 , piece_id=IGQXLHTJ22IIDFMFQTFVHO6WLQVF2H2OZ2N2QHGYFJ3URJBW3AOQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:39:45.881729333 (126h31m21.234514472s) [noise=true , address=104.36.82.55:4223 , piece_id=75KSFJRWPIELGCS7HESLVYOGDWSB3DNFLNJVMGE45F4ZZCZXF2XQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:104.36.82.55:4223): protocol: write tcp 5.161.89.252:8656->104.36.82.55:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:39:45.881799976 (126h31m21.234585104s) [noise=true , address=104.36.82.4:4818 , piece_id=X5PGQAM3LWG54G2XXY4XYDZV5UYON636CHPDFGMZRQPP4A7SWPWA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: ecclient: upload failed (node:1235eFkz7CFwwiM8UUZU1vnqDneEic7RpFzeTU46gvTMZWhhox8, address:104.36.82.4:4818): protocol: rpc: dial tcp 104.36.82.4:4818: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:39:45.881852344 (126h31m21.234637472s) [noise=true , address=104.36.82.20:4458 , piece_id=PXU2IWGZD3MNTSSZ6EUCQIEPYYAKCEUJDCLV6WM6SPRLASTLHUOA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: ecclient: upload failed (node:12SEsKJVwuQV1Z6s6doBrqsST1KhqbBREjwFYeEpME7gHExaG5d, address:104.36.82.20:4458): protocol: write tcp 5.161.89.252:9098->104.36.82.20:4458: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:39:45.881887942 (126h31m21.23467307s) [noise=true , address=104.36.82.10:4103 , piece_id=NR2RNEMTPDTOKRDFBKSEXT4IDC2HBNO5NIBVKMCYCRXXOMAABM3Q, seg_pos={0 0}, upload=111540]: Done uploading piece. err: ecclient: upload failed (node:125eQFBenzeVGCxBzzDUQDNydyAntAWGWcq38CXqjmVt3qG5Qbn, address:104.36.82.10:4103): protocol: write tcp 5.161.89.252:40448->104.36.82.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:39:45.881918068 (126h31m21.234703206s) [noise=true , address=104.36.82.44:4893 , piece_id=ACUTIIWFPHOWUNP6FNAWNKSK62WAZV6HXZC3SC645S4WJB5KKAUA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: rpc: dial tcp 104.36.82.44:4893: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:39:45.881973732 (126h31m21.234758861s) [noise=true , address=104.36.82.56:4229 , piece_id=U5RTJWHM2PCHGSCGUU5XYFP6CZWC7J7VP6INSZXWKXIX5RBFSMIA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:104.36.82.56:4229): protocol: write tcp 5.161.89.252:42454->104.36.82.56:4229: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:39:45.882039366 (126h31m21.234824494s) [seg_pos={0 0}, upload=111540]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 3 20:39:45.896234929 (126h31m21.249020058s) [upload=111540]: Done uploading segments... Sep 3 20:39:45.896252041 (126h31m21.24903717s) [upload=111540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.132641543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................................X....XXXX.X] duration: 1m0.016238514s FAIL (2025-09-03 20:39:45 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== SegmentBegin Sep 3 20:38:45.917755653 (126h30m21.270540761s) [seg_pos={0 0}, upload=111540]: Begin upload segment... Sep 3 20:38:45.917803734 (126h30m21.270588832s) [seg_pos={0 0}, upload=111540]: Done begin upload segment. Sep 3 20:38:45.917825805 (126h30m21.270610903s) [noise=true , address=104.36.82.55:4223 , piece_id=75KSFJRWPIELGCS7HESLVYOGDWSB3DNFLNJVMGE45F4ZZCZXF2XQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.917835584 (126h30m21.270620681s) [noise=true , address=104.36.82.4:4818 , piece_id=X5PGQAM3LWG54G2XXY4XYDZV5UYON636CHPDFGMZRQPP4A7SWPWA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.917907188 (126h30m21.270692287s) [noise=true , address=104.36.82.62:4436 , piece_id=3IXS5X7WSNNVJ7APQOOIOL4YCQB7S4C5BJEVWWNOZJGVQIYKA65A, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918164122 (126h30m21.27094923s) [noise=true , address=79.127.205.245:4227 , piece_id=WTVDUD4FUM5F2NGVCOM2DGJSB7JCFOAO66GJVFD6DUA6U27MRPTQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918177106 (126h30m21.270962204s) [noise=true , address=104.36.82.20:4458 , piece_id=PXU2IWGZD3MNTSSZ6EUCQIEPYYAKCEUJDCLV6WM6SPRLASTLHUOA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918463093 (126h30m21.271248201s) [noise=true , address=77.90.31.13:4640 , piece_id=3IODZBJFSVVGFAJQZWDRXGZZSSD6TZKOPZ5UX2JBIEINN45R2INA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918558682 (126h30m21.271343791s) [noise=true , address=145.79.227.2:4118 , piece_id=UCZIMO7GNDYOR4VBQTK5G4O7I72MFGF6VGX2TXVSLDCCRNHBMNKQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918709266 (126h30m21.271494364s) [noise=true , address=104.36.82.44:4893 , piece_id=ACUTIIWFPHOWUNP6FNAWNKSK62WAZV6HXZC3SC645S4WJB5KKAUA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918930031 (126h30m21.271715129s) [noise=true , address=104.36.82.61:4388 , piece_id=UKZLF3QHDC7PV3QN3DNWZA6ZNX5LOJJTAE2ZTJWHSMLKQ6XO6FSA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.918979534 (126h30m21.271764642s) [noise=true , address=104.36.82.56:4229 , piece_id=U5RTJWHM2PCHGSCGUU5XYFP6CZWC7J7VP6INSZXWKXIX5RBFSMIA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.919267074 (126h30m21.272052183s) [noise=true , address=104.36.82.60:4364 , piece_id=F6XATTDQKD6UNPZ36IHNV6BNP2FCZJOMGOVPE7HJHO5DM7W6ARZQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.919536050 (126h30m21.272321158s) [noise=true , address=104.36.82.10:4103 , piece_id=NR2RNEMTPDTOKRDFBKSEXT4IDC2HBNO5NIBVKMCYCRXXOMAABM3Q, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.919611963 (126h30m21.272397071s) [noise=true , address=45.91.116.7:4013 , piece_id=BMWM5FQ2VADDLBLMRU3DPC2CDEJCPIOAMXP7XCQFVGHCBJDUFIXA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.919772324 (126h30m21.272557422s) [noise=true , address=77.90.31.14:4692 , piece_id=3ZW7EUFU37VUYGXLT47MXKX6V5JHINNW4CVUTRSTIQBUPHCZWZTQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.919994261 (126h30m21.272779369s) [noise=true , address=104.36.82.23:4465 , piece_id=V7RWRCEEU5MCUZH26DZ3F6S7EDWDQ7QAMVNLY52J3KAMRGBOPP5A, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.920030039 (126h30m21.272815147s) [noise=true , address=77.90.31.16:4763 , piece_id=4AXJUIVFMYTFZT623PFX66JB2C4IT253XBW73F2R2EJY4KFPAODA, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.920258829 (126h30m21.273043937s) [noise=true , address=104.36.82.12:4164 , piece_id=D3B3V2LFGLMKJH35G2YR3LLW6HJGVIWPDW2L4I2M6OAYFFDPAOBQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.920499361 (126h30m21.27328447s) [noise=true , address=64.49.13.27:4961 , piece_id=IGQXLHTJ22IIDFMFQTFVHO6WLQVF2H2OZ2N2QHGYFJ3URJBW3AOQ, seg_pos={0 0}, upload=111540]: Uploading piece... Sep 3 20:38:45.994181494 (126h30m21.346966591s) [upload=111540]: Next returned nil segment Sep 3 20:38:45.994202032 (126h30m21.34698714s) [upload=111540]: Waiting for error group managing segments... Sep 3 20:38:46.014168995 (126h30m21.366954103s) [noise=true , address=104.36.82.62:4436 , piece_id=3IXS5X7WSNNVJ7APQOOIOL4YCQB7S4C5BJEVWWNOZJGVQIYKA65A, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.017977745 (126h30m21.370762842s) [noise=true , address=77.90.31.16:4763 , piece_id=4AXJUIVFMYTFZT623PFX66JB2C4IT253XBW73F2R2EJY4KFPAODA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.020311932 (126h30m21.37309704s) [noise=true , address=104.36.82.61:4388 , piece_id=UKZLF3QHDC7PV3QN3DNWZA6ZNX5LOJJTAE2ZTJWHSMLKQ6XO6FSA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.028096806 (126h30m21.380881914s) [noise=true , address=104.36.82.60:4364 , piece_id=F6XATTDQKD6UNPZ36IHNV6BNP2FCZJOMGOVPE7HJHO5DM7W6ARZQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.030036893 (126h30m21.38282199s) [noise=true , address=145.79.227.2:4118 , piece_id=UCZIMO7GNDYOR4VBQTK5G4O7I72MFGF6VGX2TXVSLDCCRNHBMNKQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.034361702 (126h30m21.387146799s) [noise=true , address=45.91.116.7:4013 , piece_id=BMWM5FQ2VADDLBLMRU3DPC2CDEJCPIOAMXP7XCQFVGHCBJDUFIXA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.042340099 (126h30m21.395125207s) [noise=true , address=77.90.31.13:4640 , piece_id=3IODZBJFSVVGFAJQZWDRXGZZSSD6TZKOPZ5UX2JBIEINN45R2INA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.047800253 (126h30m21.400585351s) [noise=true , address=104.36.82.23:4465 , piece_id=V7RWRCEEU5MCUZH26DZ3F6S7EDWDQ7QAMVNLY52J3KAMRGBOPP5A, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.048716665 (126h30m21.401501763s) [noise=true , address=77.90.31.14:4692 , piece_id=3ZW7EUFU37VUYGXLT47MXKX6V5JHINNW4CVUTRSTIQBUPHCZWZTQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.126456876 (126h30m21.479241974s) [noise=true , address=104.36.82.12:4164 , piece_id=D3B3V2LFGLMKJH35G2YR3LLW6HJGVIWPDW2L4I2M6OAYFFDPAOBQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.446267163 (126h30m21.799052291s) [noise=true , address=79.127.205.245:4227 , piece_id=WTVDUD4FUM5F2NGVCOM2DGJSB7JCFOAO66GJVFD6DUA6U27MRPTQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:38:46.539567192 (126h30m21.8923523s) [noise=true , address=64.49.13.27:4961 , piece_id=IGQXLHTJ22IIDFMFQTFVHO6WLQVF2H2OZ2N2QHGYFJ3URJBW3AOQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: <nil> Sep 3 20:39:45.881729333 (126h31m21.234514472s) [noise=true , address=104.36.82.55:4223 , piece_id=75KSFJRWPIELGCS7HESLVYOGDWSB3DNFLNJVMGE45F4ZZCZXF2XQ, seg_pos={0 0}, upload=111540]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:104.36.82.55:4223): protocol: write tcp 5.161.89.252:8656->104.36.82.55:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:39:45.881799976 (126h31m21.234585104s) [noise=true , address=104.36.82.4:4818 , piece_id=X5PGQAM3LWG54G2XXY4XYDZV5UYON636CHPDFGMZRQPP4A7SWPWA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: ecclient: upload failed (node:1235eFkz7CFwwiM8UUZU1vnqDneEic7RpFzeTU46gvTMZWhhox8, address:104.36.82.4:4818): protocol: rpc: dial tcp 104.36.82.4:4818: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:39:45.881852344 (126h31m21.234637472s) [noise=true , address=104.36.82.20:4458 , piece_id=PXU2IWGZD3MNTSSZ6EUCQIEPYYAKCEUJDCLV6WM6SPRLASTLHUOA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: ecclient: upload failed (node:12SEsKJVwuQV1Z6s6doBrqsST1KhqbBREjwFYeEpME7gHExaG5d, address:104.36.82.20:4458): protocol: write tcp 5.161.89.252:9098->104.36.82.20:4458: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:39:45.881887942 (126h31m21.23467307s) [noise=true , address=104.36.82.10:4103 , piece_id=NR2RNEMTPDTOKRDFBKSEXT4IDC2HBNO5NIBVKMCYCRXXOMAABM3Q, seg_pos={0 0}, upload=111540]: Done uploading piece. err: ecclient: upload failed (node:125eQFBenzeVGCxBzzDUQDNydyAntAWGWcq38CXqjmVt3qG5Qbn, address:104.36.82.10:4103): protocol: write tcp 5.161.89.252:40448->104.36.82.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:39:45.881918068 (126h31m21.234703206s) [noise=true , address=104.36.82.44:4893 , piece_id=ACUTIIWFPHOWUNP6FNAWNKSK62WAZV6HXZC3SC645S4WJB5KKAUA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: rpc: dial tcp 104.36.82.44:4893: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:39:45.881973732 (126h31m21.234758861s) [noise=true , address=104.36.82.56:4229 , piece_id=U5RTJWHM2PCHGSCGUU5XYFP6CZWC7J7VP6INSZXWKXIX5RBFSMIA, seg_pos={0 0}, upload=111540]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:104.36.82.56:4229): protocol: write tcp 5.161.89.252:42454->104.36.82.56:4229: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:39:45.882039366 (126h31m21.234824494s) [seg_pos={0 0}, upload=111540]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 3 20:39:45.896234929 (126h31m21.249020058s) [upload=111540]: Done uploading segments... Sep 3 20:39:45.896252041 (126h31m21.24903717s) [upload=111540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.016262919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.X..X.....................................................] duration: 463.211083ms FAIL (2025-09-03 20:39:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 463.230078ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X.................] duration: 449.242884ms FAIL (2025-09-03 20:39:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 449.262309ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.X..X....................................................] duration: 451.662494ms FAIL (2025-09-03 20:38:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments... Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment... Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment. Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece... Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments... Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil> Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments... Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 451.693431ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................X....XXXX.] duration: 1.202166929s FAIL (2025-09-03 20:38:08 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== eg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.807665219 (126h28m28.160450347s) [noise=true , address=104.36.82.56:4257 , piece_id=MU6GGTYTAFNLB4N6YY3SFTDWDCLHEEUY3P2RSGON4VAA4PKTFM3A, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.808125664 (126h28m28.160910793s) [noise=true , address=104.36.82.35:4612 , piece_id=TJKEU2IYCTVT4KZQVENPIVOQF6LLBMEGEXB4ZNKEBLIHIXNVSGVQ, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.808144079 (126h28m28.160929197s) [noise=true , address=79.127.219.47:4052 , piece_id=NH6JVWJAFBZQ2E54MIOU4QVJCF7HKOTN5YYMH6D5PWJ6I4VOCKLA, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.808503735 (126h28m28.161288864s) [noise=true , address=104.36.82.9:4019 , piece_id=W53AUZUV34XLNT5IHNYQ6EJ6ILL7QDREUCRU2JQDG7DYM6AXK2HA, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.808518483 (126h28m28.161303611s) [noise=true , address=104.36.82.61:4388 , piece_id=2KSZL4J3656GB6O3DUNLJPBKUTRM2OEC2UA4B3GVFAC2ETBENLGQ, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.808865996 (126h28m28.161651114s) [noise=true , address=104.36.82.58:4299 , piece_id=BZQKQW27VMEVCJD62KCEV5NLPZQGQAONHVZM6BMG5TV3BAZIVISA, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.808890322 (126h28m28.16167544s) [noise=true , address=104.36.82.23:4480 , piece_id=4YDIUNDJABZJFYY2V7ZIJM2VPE3UXVA3KA3A6BPT4OBW4THHHMYQ, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.809233177 (126h28m28.162018305s) [noise=true , address=104.36.82.48:4594 , piece_id=OSQJ4H27DYNR34SBYGN7J7WZAH76O4YYIGIPPKWMDFKX7CAWF3YQ, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.809265027 (126h28m28.162050144s) [noise=true , address=45.91.116.5:4317 , piece_id=EHY2BYYR2HZG5D7TU7N33IIUD3UGMYQUUPGMITGOOXNGBHREY6VQ, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.809619413 (126h28m28.162404531s) [noise=true , address=104.36.82.50:4047 , piece_id=A6D3KOFCO4FJK364UNGHTTGRSEQSTLC4P26MPAR3M2TGJNZJUMQQ, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.809629241 (126h28m28.16241437s) [noise=true , address=77.90.31.14:4680 , piece_id=PO3EB6YTTCK64XQIFCYHTO4MDLTYQ6CTAVDMKI4SMXA5H7X5TWGQ, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.810012462 (126h28m28.162797589s) [noise=true , address=104.36.82.34:4564 , piece_id=GA466AU63BKOBE2MMKBP7OCQRGJUSSTPIDYZBXCRRW5435OLLLWQ, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.810062735 (126h28m28.162847864s) [noise=true , address=77.90.31.15:4717 , piece_id=MZIHOOUWN77ZBVFGCYV5JWIBQUJMTTHEGU3DHMUFI7RN7ZBKVZKQ, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.810394029 (126h28m28.163179157s) [noise=true , address=104.36.82.54:4178 , piece_id=6QWPP6EJ4BDPOKN2EH6X2VCOO2C57T4WFKKQEE2MUTY7KWHX5SJA, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.810553499 (126h28m28.163338627s) [noise=true , address=104.36.82.55:4210 , piece_id=3QAD2MMEJX6MTFQNRZIXI3CMXEAYUP3W4RLDEVCA45KYA2CNOQAA, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.811087092 (126h28m28.163872219s) [noise=true , address=104.36.82.62:4437 , piece_id=J4HVOQ5J4LPEEIHNRA72TJW5PKU6TV77S6DPQWY5TCIPHTGCAHGQ, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.813296114 (126h28m28.166081242s) [noise=true , address=104.36.82.60:4355 , piece_id=V3CXQX6UWQXP4M6EX3MJKKGXNVX43KX7QDXGQC6VL5WQGISAIGQQ, seg_pos={0 0}, upload=111514]: Uploading piece... Sep 3 20:36:52.900114148 (126h28m28.252899266s) [upload=111514]: Next returned nil segment Sep 3 20:36:52.900127864 (126h28m28.252912982s) [upload=111514]: Waiting for error group managing segments... Sep 3 20:36:52.932379749 (126h28m28.285164878s) [noise=true , address=77.90.31.14:4680 , piece_id=PO3EB6YTTCK64XQIFCYHTO4MDLTYQ6CTAVDMKI4SMXA5H7X5TWGQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil> Sep 3 20:36:52.941719876 (126h28m28.294505014s) [noise=true , address=79.127.219.47:4052 , piece_id=NH6JVWJAFBZQ2E54MIOU4QVJCF7HKOTN5YYMH6D5PWJ6I4VOCKLA, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil> Sep 3 20:36:52.943004581 (126h28m28.295789709s) [noise=true , address=104.36.82.58:4299 , piece_id=BZQKQW27VMEVCJD62KCEV5NLPZQGQAONHVZM6BMG5TV3BAZIVISA, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil> Sep 3 20:36:52.948243579 (126h28m28.301028707s) [noise=true , address=77.90.31.15:4717 , piece_id=MZIHOOUWN77ZBVFGCYV5JWIBQUJMTTHEGU3DHMUFI7RN7ZBKVZKQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil> Sep 3 20:36:52.951124274 (126h28m28.303909412s) [noise=true , address=104.36.82.61:4388 , piece_id=2KSZL4J3656GB6O3DUNLJPBKUTRM2OEC2UA4B3GVFAC2ETBENLGQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil> Sep 3 20:36:52.967611516 (126h28m28.320396643s) [noise=true , address=45.91.116.5:4317 , piece_id=EHY2BYYR2HZG5D7TU7N33IIUD3UGMYQUUPGMITGOOXNGBHREY6VQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil> Sep 3 20:36:52.980651166 (126h28m28.333436295s) [noise=true , address=104.36.82.62:4437 , piece_id=J4HVOQ5J4LPEEIHNRA72TJW5PKU6TV77S6DPQWY5TCIPHTGCAHGQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil> Sep 3 20:36:52.999845226 (126h28m28.352630354s) [noise=true , address=104.36.82.54:4178 , piece_id=6QWPP6EJ4BDPOKN2EH6X2VCOO2C57T4WFKKQEE2MUTY7KWHX5SJA, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil> Sep 3 20:36:53.111093592 (126h28m28.463878731s) [noise=true , address=104.36.82.23:4480 , piece_id=4YDIUNDJABZJFYY2V7ZIJM2VPE3UXVA3KA3A6BPT4OBW4THHHMYQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil> Sep 3 20:36:53.556461523 (126h28m28.909246652s) [noise=true , address=104.36.82.60:4355 , piece_id=V3CXQX6UWQXP4M6EX3MJKKGXNVX43KX7QDXGQC6VL5WQGISAIGQQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil> Sep 3 20:37:52.770263630 (126h29m28.123048738s) [noise=true , address=104.36.82.56:4257 , piece_id=MU6GGTYTAFNLB4N6YY3SFTDWDCLHEEUY3P2RSGON4VAA4PKTFM3A, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:12iw1KEPBAYhvSachvkigv7bPXfGN5XEugvxGC5YTBV2Muymir3, address:104.36.82.56:4257): protocol: rpc: dial tcp 104.36.82.56:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:52.770305489 (126h29m28.123090597s) [noise=true , address=104.36.82.48:4594 , piece_id=OSQJ4H27DYNR34SBYGN7J7WZAH76O4YYIGIPPKWMDFKX7CAWF3YQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:12ACMz4vh1qJLsKN5FkjNgitVHjwmjSBJ7rF77h2NF41bQJUaPr, address:104.36.82.48:4594): protocol: rpc: dial tcp 104.36.82.48:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:52.770372094 (126h29m28.123157202s) [noise=true , address=104.36.82.50:4047 , piece_id=A6D3KOFCO4FJK364UNGHTTGRSEQSTLC4P26MPAR3M2TGJNZJUMQQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: write tcp 5.161.89.252:48586->104.36.82.50:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:52.770410878 (126h29m28.123195975s) [noise=true , address=104.36.82.51:4058 , piece_id=36TJOSLTKYEOOHHS42HABQQOSJBRJZ5R6QUFCQMZ2EC47TH5H7HQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: rpc: dial tcp 104.36.82.51:4058: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:52.770460691 (126h29m28.123245799s) [noise=true , address=104.36.82.35:4612 , piece_id=TJKEU2IYCTVT4KZQVENPIVOQF6LLBMEGEXB4ZNKEBLIHIXNVSGVQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: rpc: dial tcp 104.36.82.35:4612: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:52.770491839 (126h29m28.123276947s) [noise=true , address=104.36.82.9:4019 , piece_id=W53AUZUV34XLNT5IHNYQ6EJ6ILL7QDREUCRU2JQDG7DYM6AXK2HA, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:12JuQKLQnXiEcyCj6xthEXSPy2Wu5T1B9m4q9YteoK69rMqVydD, address:104.36.82.9:4019): protocol: rpc: dial tcp 104.36.82.9:4019: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:52.770500074 (126h29m28.123285183s) [noise=true , address=104.36.82.55:4210 , piece_id=3QAD2MMEJX6MTFQNRZIXI3CMXEAYUP3W4RLDEVCA45KYA2CNOQAA, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: rpc: dial tcp 104.36.82.55:4210: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:52.770527787 (126h29m28.123312885s) [noise=true , address=104.36.82.34:4564 , piece_id=GA466AU63BKOBE2MMKBP7OCQRGJUSSTPIDYZBXCRRW5435OLLLWQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:104.36.82.34:4564): protocol: write tcp 5.161.89.252:46828->104.36.82.34:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:52.770556220 (126h29m28.123341328s) [seg_pos={0 0}, upload=111514]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Sep 3 20:37:52.785614907 (126h29m28.138400015s) [upload=111514]: Done uploading segments... Sep 3 20:37:52.785629805 (126h29m28.138414902s) [upload=111514]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1.202200413s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X................] duration: 451.605376ms FAIL (2025-09-03 20:38:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 451.623449ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)