Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XXXX.X.X..................................................] duration: 385.726153ms FAIL (2025-09-03 21:19:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 385.759767ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XXXX.X.X.................................................] duration: 394.540757ms FAIL (2025-09-03 21:18:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 394.566476ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................XXX........................................] duration: 381.907815ms FAIL (2025-09-03 21:18:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 381.963489ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................XXX.......................................] duration: 473.261489ms FAIL (2025-09-03 21:17:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 473.281156ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XXXX.X.X................................................] duration: 801.07059ms FAIL (2025-09-03 21:17:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 801.103141ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................XXX......................................] duration: 717.409116ms FAIL (2025-09-03 21:16:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 717.434034ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [X....XXXX.X.X...............................................] duration: 596.591261ms FAIL (2025-09-03 21:16:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 596.627228ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................XXX.....................................] duration: 451.349273ms FAIL (2025-09-03 21:16:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 451.366555ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X....XXXX.X.X..............................................] duration: 855.518383ms FAIL (2025-09-03 21:15:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 855.55876ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................XXX....................................] duration: 605.945068ms FAIL (2025-09-03 21:15:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 605.970306ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X....XXXX.X.X.............................................] duration: 416.167842ms FAIL (2025-09-03 21:15:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 416.194172ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................XXX...................................] duration: 487.34944ms FAIL (2025-09-03 21:14:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.372504ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X....XXXX.X.X............................................] duration: 531.549137ms FAIL (2025-09-03 21:13:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 531.568473ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X....XXXX.X.X...........................................] duration: 421.606839ms FAIL (2025-09-03 21:13:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 421.631997ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................XXX..................................] duration: 410.994805ms FAIL (2025-09-03 21:12:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 411.006577ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................XXX.................................] duration: 754.96011ms FAIL (2025-09-03 21:12:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 754.98135ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X....XXXX.X.X..........................................] duration: 445.986568ms FAIL (2025-09-03 21:12:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 446.010163ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................XXX................................] duration: 461.540032ms FAIL (2025-09-03 21:11:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 461.559969ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X....XXXX.X.X.........................................] duration: 383.068537ms FAIL (2025-09-03 21:11:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 383.092252ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................XXX...............................] duration: 553.957137ms FAIL (2025-09-03 21:11:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 553.974039ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................XXX..............................] duration: 390.554753ms FAIL (2025-09-03 21:10:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 390.667385ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X....XXXX.X.X........................................] duration: 953.321366ms FAIL (2025-09-03 21:10:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 953.342866ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...........................XXX.............................] duration: 568.379367ms FAIL (2025-09-03 21:10:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 568.405484ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X....XXXX.X.X.......................................] duration: 523.330964ms FAIL (2025-09-03 21:10:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 523.353338ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...........................XXX............................] duration: 422.156041ms FAIL (2025-09-03 21:09:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 422.177551ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X....XXXX.X.X......................................] duration: 647.463914ms FAIL (2025-09-03 21:08:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 647.490483ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...........................XXX...........................] duration: 983.456477ms FAIL (2025-09-03 21:08:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 983.47388ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X....XXXX.X.X.....................................] duration: 865.499381ms FAIL (2025-09-03 21:08:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 865.518817ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X...........................XXX..........................] duration: 1.229283439s FAIL (2025-09-03 21:08:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.229299418s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X....XXXX.X.X....................................] duration: 1.499100799s FAIL (2025-09-03 21:07:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.499123982s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X...........................XXX.........................] duration: 395.857635ms FAIL (2025-09-03 21:07:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 395.869105ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X...........................XXX........................] duration: 413.588454ms FAIL (2025-09-03 21:07:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 413.6138ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X....XXXX.X.X...................................] duration: 546.91352ms FAIL (2025-09-03 21:06:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 546.942535ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.....X...........................XXX.......................] duration: 2.00383371s FAIL (2025-09-03 21:06:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.003889934s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X....XXXX.X.X..................................] duration: 1.570523611s FAIL (2025-09-03 21:06:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.570546474s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX.....X...........................XXX......................] duration: 910.168563ms FAIL (2025-09-03 21:06:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 910.184884ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............X....XXXX.X.X.................................] duration: 904.550494ms FAIL (2025-09-03 21:05:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 904.580221ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX.....X...........................XXX.....................] duration: 1.712073378s FAIL (2025-09-03 21:05:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.712092073s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............X....XXXX.X.X................................] duration: 412.587836ms FAIL (2025-09-03 21:04:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 412.616969ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX.....X...........................XXX....................] duration: 494.185755ms FAIL (2025-09-03 21:04:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 494.210442ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [................X....XXXX.X.X...............................] duration: 663.32525ms FAIL (2025-09-03 21:04:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 663.36128ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XX.....X...........................XXX...................] duration: 870.493978ms FAIL (2025-09-03 21:04:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 870.514596ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XX.....X...........................XXX..................] duration: 736.389268ms FAIL (2025-09-03 21:03:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 736.407843ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................X....XXXX.X.X..............................] duration: 969.15233ms FAIL (2025-09-03 21:03:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 969.200631ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................X....XXXX.X.X.............................] duration: 759.039119ms FAIL (2025-09-03 21:02:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 759.062513ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XX.....X...........................XXX.................] duration: 1.284200563s FAIL (2025-09-03 21:02:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.284218386s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................X....XXXX.X.X............................] duration: 765.661934ms FAIL (2025-09-03 21:02:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 765.681421ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......XX.....X...........................XXX................] duration: 482.785104ms FAIL (2025-09-03 21:01:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 482.809047ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................X....XXXX.X.X...........................] duration: 730.805736ms FAIL (2025-09-03 21:01:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 730.829491ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................X....XXXX.X.X..........................] duration: 491.65424ms FAIL (2025-09-03 21:01:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 491.67584ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)