Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.245212833s FAIL (2025-10-31 22:38:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.245228171s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X........................XXXX..X..X......X.X.............] duration: 4.126068658s FAIL (2025-10-31 22:38:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.126085399s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 779.840864ms FAIL (2025-10-31 22:38:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 779.856864ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 503.438605ms FAIL (2025-10-31 22:38:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments... Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment... Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment. Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments... Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments... Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 503.46213ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X........................XXXX..X..X......X.X............] duration: 4.309315182s FAIL (2025-10-31 22:37:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.309333206s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 583.178405ms FAIL (2025-10-31 22:37:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 583.196037ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 440.154714ms FAIL (2025-10-31 22:37:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments... Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment... Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment. Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments... Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments... Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 440.176125ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 813.520436ms FAIL (2025-10-31 22:37:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 813.53398ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 691.318567ms FAIL (2025-10-31 22:37:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 691.3353ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X........................XXXX..X..X......X.X...........] duration: 16.111529396s FAIL (2025-10-31 22:36:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.111544554s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 759.630988ms FAIL (2025-10-31 22:36:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments... Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment... Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment. Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments... Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments... Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 759.648451ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 519.053387ms FAIL (2025-10-31 22:36:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 519.074305ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 693.258167ms FAIL (2025-10-31 22:36:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 693.283745ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 404.970813ms FAIL (2025-10-31 22:36:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments... Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment... Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment. Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments... Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments... Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 404.987834ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X........................XXXX..X..X......X.X..........] duration: 6.495919634s FAIL (2025-10-31 22:35:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.495937977s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 533.531906ms FAIL (2025-10-31 22:35:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 533.6403ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X........................XXXX..X..X......X.X.........] duration: 1.615204357s FAIL (2025-10-31 22:35:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.615222872s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 638.584775ms FAIL (2025-10-31 22:35:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 638.626954ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 484.986148ms FAIL (2025-10-31 22:34:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments... Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment... Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment. Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments... Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments... Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 485.010052ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 662.439502ms FAIL (2025-10-31 22:34:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 662.458025ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 447.95296ms FAIL (2025-10-31 22:34:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments... Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment... Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment. Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments... Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments... Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 447.978257ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.481182545s FAIL (2025-10-31 22:34:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.48120651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........X........................XXXX..X..X......X.X........] duration: 1.86616368s FAIL (2025-10-31 22:34:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.866181884s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 575.482771ms FAIL (2025-10-31 22:33:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 575.50345ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 403.55532ms FAIL (2025-10-31 22:33:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments... Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment... Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment. Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments... Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments... Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 403.578123ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 787.490536ms FAIL (2025-10-31 22:33:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 787.513298ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........X........................XXXX..X..X......X.X.......] duration: 911.778464ms FAIL (2025-10-31 22:33:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 911.812017ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 384.982447ms FAIL (2025-10-31 22:32:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments... Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment... Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment. Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments... Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments... Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 385.004608ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.052168542s FAIL (2025-10-31 22:32:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.052187507s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 818.662842ms FAIL (2025-10-31 22:32:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 818.681076ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 837.269074ms FAIL (2025-10-31 22:32:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 837.290975ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........X........................XXXX..X..X......X.X......] duration: 4.610077045s FAIL (2025-10-31 22:32:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.61011117s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 345.852788ms FAIL (2025-10-31 22:32:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments... Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment... Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment. Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments... Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments... Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 345.872254ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 858.817546ms FAIL (2025-10-31 22:31:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 858.842503ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 794.027892ms FAIL (2025-10-31 22:31:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 794.04244ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........X........................XXXX..X..X......X.X.....] duration: 498.801958ms FAIL (2025-10-31 22:31:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 498.820393ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 790.476141ms FAIL (2025-10-31 22:31:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 790.499626ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [............X........................XXXX..X..X......X.X....] duration: 8.429801717s FAIL (2025-10-31 22:31:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.429847492s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 852.803525ms FAIL (2025-10-31 22:31:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 852.825707ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 482.594602ms FAIL (2025-10-31 22:31:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments... Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment... Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment. Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments... Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments... Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 482.625943ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 575.033777ms FAIL (2025-10-31 22:31:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 575.053825ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............X........................XXXX..X..X......X.X...] duration: 10.536914815s FAIL (2025-10-31 22:30:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.53693853s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.175709353s FAIL (2025-10-31 22:30:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.175794472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.598690012s FAIL (2025-10-31 22:30:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments... Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment... Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment. Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments... Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments... Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.598709087s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.1723442s FAIL (2025-10-31 22:30:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.17235966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............X........................XXXX..X..X......X.X..] duration: 10.428452012s FAIL (2025-10-31 22:30:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.42846718s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 594.172602ms FAIL (2025-10-31 22:29:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 594.19312ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 422.166302ms FAIL (2025-10-31 22:29:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments... Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment... Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment. Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments... Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments... Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 422.191248ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 674.606287ms FAIL (2025-10-31 22:29:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 674.622136ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 475.662805ms FAIL (2025-10-31 22:29:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments... Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment... Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment. Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece... Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments... Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil> Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments... Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 475.683714ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)