Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................XXX................................] duration: 461.540032ms FAIL (2025-09-03 21:11:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 461.559969ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X....XXXX.X.X.........................................] duration: 383.068537ms FAIL (2025-09-03 21:11:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 383.092252ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................XXX...............................] duration: 553.957137ms FAIL (2025-09-03 21:11:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 553.974039ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................XXX..............................] duration: 390.554753ms FAIL (2025-09-03 21:10:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 390.667385ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X....XXXX.X.X........................................] duration: 953.321366ms FAIL (2025-09-03 21:10:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 953.342866ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...........................XXX.............................] duration: 568.379367ms FAIL (2025-09-03 21:10:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 568.405484ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X....XXXX.X.X.......................................] duration: 523.330964ms FAIL (2025-09-03 21:10:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 523.353338ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...........................XXX............................] duration: 422.156041ms FAIL (2025-09-03 21:09:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 422.177551ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X....XXXX.X.X......................................] duration: 647.463914ms FAIL (2025-09-03 21:08:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 647.490483ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...........................XXX...........................] duration: 983.456477ms FAIL (2025-09-03 21:08:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 983.47388ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X....XXXX.X.X.....................................] duration: 865.499381ms FAIL (2025-09-03 21:08:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 865.518817ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X...........................XXX..........................] duration: 1.229283439s FAIL (2025-09-03 21:08:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.229299418s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X....XXXX.X.X....................................] duration: 1.499100799s FAIL (2025-09-03 21:07:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.499123982s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X...........................XXX.........................] duration: 395.857635ms FAIL (2025-09-03 21:07:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 395.869105ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X...........................XXX........................] duration: 413.588454ms FAIL (2025-09-03 21:07:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 413.6138ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X....XXXX.X.X...................................] duration: 546.91352ms FAIL (2025-09-03 21:06:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 546.942535ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.....X...........................XXX.......................] duration: 2.00383371s FAIL (2025-09-03 21:06:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.003889934s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X....XXXX.X.X..................................] duration: 1.570523611s FAIL (2025-09-03 21:06:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.570546474s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX.....X...........................XXX......................] duration: 910.168563ms FAIL (2025-09-03 21:06:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 910.184884ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............X....XXXX.X.X.................................] duration: 904.550494ms FAIL (2025-09-03 21:05:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 904.580221ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX.....X...........................XXX.....................] duration: 1.712073378s FAIL (2025-09-03 21:05:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.712092073s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............X....XXXX.X.X................................] duration: 412.587836ms FAIL (2025-09-03 21:04:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 412.616969ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX.....X...........................XXX....................] duration: 494.185755ms FAIL (2025-09-03 21:04:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 494.210442ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [................X....XXXX.X.X...............................] duration: 663.32525ms FAIL (2025-09-03 21:04:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 663.36128ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XX.....X...........................XXX...................] duration: 870.493978ms FAIL (2025-09-03 21:04:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 870.514596ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XX.....X...........................XXX..................] duration: 736.389268ms FAIL (2025-09-03 21:03:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 736.407843ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................X....XXXX.X.X..............................] duration: 969.15233ms FAIL (2025-09-03 21:03:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 969.200631ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................X....XXXX.X.X.............................] duration: 759.039119ms FAIL (2025-09-03 21:02:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 759.062513ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XX.....X...........................XXX.................] duration: 1.284200563s FAIL (2025-09-03 21:02:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.284218386s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................X....XXXX.X.X............................] duration: 765.661934ms FAIL (2025-09-03 21:02:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 765.681421ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......XX.....X...........................XXX................] duration: 482.785104ms FAIL (2025-09-03 21:01:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 482.809047ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................X....XXXX.X.X...........................] duration: 730.805736ms FAIL (2025-09-03 21:01:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 730.829491ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................X....XXXX.X.X..........................] duration: 491.65424ms FAIL (2025-09-03 21:01:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 491.67584ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......XX.....X...........................XXX...............] duration: 492.356571ms FAIL (2025-09-03 21:00:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 492.399452ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................X....XXXX.X.X.........................] duration: 801.205786ms FAIL (2025-09-03 21:00:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 801.236624ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.......XX.....X...........................XXX..............] duration: 391.931137ms FAIL (2025-09-03 21:00:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 392.008242ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................X....XXXX.X.X........................] duration: 620.0069ms FAIL (2025-09-03 20:59:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 620.032508ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................X....XXXX.X.X.......................] duration: 577.489346ms FAIL (2025-09-03 20:59:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 577.514072ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.......XX.....X...........................XXX.............] duration: 496.235751ms FAIL (2025-09-03 20:59:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 496.276157ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................X....XXXX.X.X......................] duration: 406.776232ms FAIL (2025-09-03 20:58:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 406.803614ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.X.......XX.....X...........................XXX............] duration: 394.465891ms FAIL (2025-09-03 20:57:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 394.481933ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.X.......XX.....X...........................XXX...........] duration: 652.937992ms FAIL (2025-09-03 20:57:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 653.004407ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................X....XXXX.X.X.....................] duration: 779.479249ms FAIL (2025-09-03 20:57:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 779.498526ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.X.X.......XX.....X...........................XXX..........] duration: 478.977389ms FAIL (2025-09-03 20:57:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.001604ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................X....XXXX.X.X....................] duration: 730.183777ms FAIL (2025-09-03 20:56:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 730.211849ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.X.X.......XX.....X...........................XXX.........] duration: 392.247371ms FAIL (2025-09-03 20:56:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 392.264504ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................X....XXXX.X.X...................] duration: 520.217082ms FAIL (2025-09-03 20:56:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 520.235257ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.X.X.X.......XX.....X...........................XXX........] duration: 628.123523ms FAIL (2025-09-03 20:55:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 628.140135ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............................X....XXXX.X.X..................] duration: 1.143375557s FAIL (2025-09-03 20:54:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ents... Sep 3 20:40:01.672425847 (126h31m37.025210986s) [upload=111560]: Got next segment. Inline: false Sep 3 20:40:01.672488585 (126h31m37.025273724s) [upload=111560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:01.672495959 (126h31m37.025281097s) [upload=111560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:01.699801005 (126h31m37.052586132s) [seg_pos={0 0}, upload=111560]: Begin upload segment... Sep 3 20:40:01.699850838 (126h31m37.052635966s) [seg_pos={0 0}, upload=111560]: Done begin upload segment. Sep 3 20:40:01.699878521 (126h31m37.052663649s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700583896 (126h31m37.053369024s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.700767521 (126h31m37.05355265s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.701473809 (126h31m37.054258937s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.702203851 (126h31m37.05498898s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703081140 (126h31m37.055866269s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.703488978 (126h31m37.056274105s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704391934 (126h31m37.057177133s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.704915178 (126h31m37.057700306s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.705295643 (126h31m37.058080771s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706453329 (126h31m37.059238457s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.706825800 (126h31m37.059610928s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707263984 (126h31m37.060049112s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.707662922 (126h31m37.060448041s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.708794100 (126h31m37.061579237s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.712543898 (126h31m37.065329026s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.713135359 (126h31m37.065920488s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.714219027 (126h31m37.067004155s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Uploading piece... Sep 3 20:40:01.806369676 (126h31m37.159154773s) [upload=111560]: Next returned nil segment Sep 3 20:40:01.806381477 (126h31m37.159166576s) [upload=111560]: Waiting for error group managing segments... Sep 3 20:40:01.822572051 (126h31m37.175357159s) [noise=true , address=104.36.82.23:4476 , piece_id=PIQO2Z3AINMGP5OJDSL4XHN6I45AJKCVSRWUINZRGS3DF6XJWT7A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.844806947 (126h31m37.197592055s) [noise=true , address=45.91.116.4:4956 , piece_id=NO22CXIUN2FFBPAFJNQDASJOXCR2534EBQHF3LW4U2AOXU7ZB6XA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.846763445 (126h31m37.199548543s) [noise=true , address=104.36.82.61:4394 , piece_id=URVQF7ADOBL3YUUUTJMMRDFLLTUGMXFGY7ZU6QOFWP7KUH4ONEUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.852773823 (126h31m37.20555892s) [noise=true , address=77.90.31.15:4722 , piece_id=26SLRVA425SIHSDN5ME7AP6Q6RJJCRA34AHFVVHRUT3OCQOZDKXA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.857112999 (126h31m37.209898097s) [noise=true , address=104.36.82.58:4269 , piece_id=3SUPE7WILEJZF4AQKFNG7H67NQLRVM5HFXYA7ODRT4GJXC4MENEQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.892915358 (126h31m37.245700466s) [noise=true , address=45.91.116.10:4111 , piece_id=GTVFQYIR3K4DB5FFON2RLJHSITZJR4H7QB4FUB7TXTYVBPTYLX4Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.912506614 (126h31m37.265291722s) [noise=true , address=104.36.82.12:4174 , piece_id=GFFUYYIDCP4YKMGKZIINO5IYCXLUXTXE3FJNM7MOYNL7PBWTVHUQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:01.935474448 (126h31m37.288259556s) [noise=true , address=104.36.82.62:4429 , piece_id=PLAVD3I7AVEFANTNZ6FPAQWGELKZWQTANWYQ2OGITMOK4NISM65A, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.006631334 (126h31m37.359416432s) [noise=true , address=77.90.31.18:4826 , piece_id=FBONN27GYA6EBACELYR3APITXIL6EDJUMUNUDZPKRFEUHHUFENIA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.064030909 (126h31m37.416816007s) [noise=true , address=77.90.31.10:4550 , piece_id=7FKBX2ZBAS5R5L6VOTVZWWRTX7CVPNNTU6II3YASKL4JPDNTVJJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.261769685 (126h31m37.614554793s) [noise=true , address=104.36.82.59:4306 , piece_id=UKKBMJ3Z7PMDI5GAE7TDCCXRWUF77TRXXZYLPX2D6G44UY2TKX5Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.318333911 (126h31m37.671119008s) [noise=true , address=64.49.13.137:4501 , piece_id=3DZ4LIPZ643UAQL7WEOBX7SV2GLWKU7I7CELZ4OWO4UEF2PNRYPQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:40:02.366567176 (126h31m37.719352274s) [noise=true , address=45.91.116.5:4322 , piece_id=2X23WDWSREKUDIN5GLKTGXWWMVRNEBTO4PEMIJ4PIXMJMZOLZKJA, seg_pos={0 0}, upload=111560]: Done uploading piece. err: <nil> Sep 3 20:41:01.657584095 (126h32m37.010369233s) [noise=true , address=104.36.82.5:4846 , piece_id=4CH2EYRBG7F3RXE6CORK43B6XCK7EHRI3A27KCMFVAOINXHYTYAQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657678192 (126h32m37.010463329s) [noise=true , address=104.36.82.51:4062 , piece_id=7X35ES73HJD33R7NNRYMA74KJODGMRFA7XL5TDH2BJIP7K2TSK2Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657729148 (126h32m37.010514286s) [noise=true , address=104.36.82.53:4155 , piece_id=7BQTMMI2WYFOWKGYBWXMDFTMWFHKORFCJY6BDXEHUY5IVPP63Y7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657798749 (126h32m37.010583876s) [noise=true , address=104.36.82.49:4985 , piece_id=RENWIXUYNSZ62PVEN5NCXDKUR72ZKLE5OBTB7NBX2Q3ZCAQCLVGQ, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657834856 (126h32m37.010619984s) [noise=true , address=104.36.82.51:4058 , piece_id=DHDBTAAJ4DJYY4J3ZU55SFXQ7M4D3X3AXZUTPY2C3VNX6L4FKK7Q, seg_pos={0 0}, upload=111560]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: write tcp 5.161.89.252:51936->104.36.82.51:4058: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:01.657874561 (126h32m37.010659689s) [seg_pos={0 0}, upload=111560]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:01.674900145 (126h32m37.027685274s) [upload=111560]: Done uploading segments... Sep 3 20:41:01.674914953 (126h32m37.027700081s) [upload=111560]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.14341925s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.X.X.X.......XX.....X...........................XXX.......] duration: 530.049268ms FAIL (2025-09-03 20:54:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 530.06561ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)