Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................X.............................X.............] duration:
404.288226ms
FAIL (2025-10-31 22:02:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 404.339341ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.......X.......X.................................X..........] duration:
488.206696ms
FAIL (2025-10-31 22:02:19 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 488.247922ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........X.X....X..........................................] duration:
435.261045ms
FAIL (2025-10-31 22:01:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 435.277996ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [......X.......X.X...............................X...........] duration:
610.82999ms
FAIL (2025-10-31 22:01:58 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 610.843166ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [........X.......X.................................X.........] duration:
546.163304ms
FAIL (2025-10-31 22:01:49 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 546.189843ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X........X.................................X............] duration:
438.436919ms
FAIL (2025-10-31 22:01:30 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 438.450074ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X..........X.X....X.........................................] duration:
348.98189ms
FAIL (2025-10-31 22:01:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 349.007658ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X................X.............................X............] duration:
336.689011ms
FAIL (2025-10-31 22:01:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 336.706775ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.........X.......X.................................X........] duration:
508.620955ms
FAIL (2025-10-31 22:01:19 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 508.702799ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.......X.......X.X...............................X..........] duration:
669.24037ms
FAIL (2025-10-31 22:01:06 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 669.258554ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X........X.................................X...........] duration:
384.929297ms
FAIL (2025-10-31 22:01:00 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 384.95773ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X................X.............................X...........] duration:
495.713371ms
FAIL (2025-10-31 22:00:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 495.731444ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [..........X.......X.................................X.......] duration:
507.568502ms
FAIL (2025-10-31 22:00:51 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 507.621211ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X........X.................................X..........] duration:
366.073539ms
FAIL (2025-10-31 22:00:35 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 366.099467ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X..........X.X....X........................................] duration:
356.242447ms
FAIL (2025-10-31 22:00:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 356.268265ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [........X.......X.X...............................X.........] duration:
664.111441ms
FAIL (2025-10-31 22:00:34 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 664.130567ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X........X.................................X.........] duration:
398.847797ms
FAIL (2025-10-31 22:00:19 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 398.866491ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X................X.............................X..........] duration:
416.311091ms
FAIL (2025-10-31 22:00:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 416.324406ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X........X.................................X........] duration:
382.450108ms
FAIL (2025-10-31 21:59:53 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 382.470136ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X..........X.X....X.......................................] duration:
477.458212ms
FAIL (2025-10-31 21:59:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 477.486413ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.........X.......X.X...............................X........] duration:
758.234971ms
FAIL (2025-10-31 21:59:48 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 758.259236ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [...........X.......X.................................X......] duration:
641.433819ms
FAIL (2025-10-31 21:59:41 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 641.456361ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..........X.......X.X...............................X.......] duration:
625.598734ms
FAIL (2025-10-31 21:59:32 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 625.615716ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X..........X.X....X......................................] duration:
477.244731ms
FAIL (2025-10-31 21:59:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 477.2651ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X................X.............................X.........] duration:
377.219894ms
FAIL (2025-10-31 21:59:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 377.235614ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X........X.................................X.......] duration:
432.855633ms
FAIL (2025-10-31 21:59:21 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 432.887543ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X................X.............................X........] duration:
403.690395ms
FAIL (2025-10-31 21:59:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 403.704132ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........X........X.................................X......] duration:
351.070136ms
FAIL (2025-10-31 21:59:03 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 351.096805ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X..........X.X....X.....................................] duration:
479.439165ms
FAIL (2025-10-31 21:58:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 479.466025ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X................X.............................X.......] duration:
353.01465ms
FAIL (2025-10-31 21:58:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 353.035479ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...........X.......X.X...............................X......] duration:
911.02605ms
FAIL (2025-10-31 21:58:40 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 911.045036ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=3) [X...........X.......X.................................X.....] duration:
495.564534ms
FAIL (2025-10-31 21:58:29 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 495.579281ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [............X.......X.X...............................X.....] duration:
635.450137ms
FAIL (2025-10-31 21:58:20 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 635.503146ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........X........X.................................X.....] duration:
410.447085ms
FAIL (2025-10-31 21:58:07 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 410.474035ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X..........X.X....X....................................] duration:
450.915209ms
FAIL (2025-10-31 21:57:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 450.938073ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X................X.............................X......] duration:
354.275555ms
FAIL (2025-10-31 21:57:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 354.299851ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=3) [.X...........X.......X.................................X....] duration:
658.587351ms
FAIL (2025-10-31 21:57:38 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 658.601637ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.............X.......X.X...............................X....] duration:
609.467777ms
FAIL (2025-10-31 21:57:34 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 609.489258ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X................X.............................X.....] duration:
385.787022ms
FAIL (2025-10-31 21:57:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 385.807571ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............X........X.................................X....] duration:
372.904359ms
FAIL (2025-10-31 21:57:09 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 372.922283ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X..........X.X....X...................................] duration:
573.457255ms
FAIL (2025-10-31 21:56:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 573.482953ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=3) [..X...........X.......X.................................X...] duration:
496.706133ms
FAIL (2025-10-31 21:56:40 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 496.72559ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..............X.......X.X...............................X...] duration:
5.105741098s
FAIL (2025-10-31 21:56:39 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.105755635s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............X........X.................................X...] duration:
373.098541ms
FAIL (2025-10-31 21:56:22 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 373.378395ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X................X.............................X....] duration:
344.306142ms
FAIL (2025-10-31 21:56:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 344.329115ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...............X.......X.X...............................X..] duration:
703.001204ms
FAIL (2025-10-31 21:56:10 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 703.027133ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=3) [...X...........X.......X.................................X..] duration:
601.15071ms
FAIL (2025-10-31 21:56:04 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 601.172862ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............X........X.................................X..] duration:
370.53785ms
FAIL (2025-10-31 21:55:43 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 370.558479ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X..........X.X....X..................................] duration:
5.913979418s
FAIL (2025-10-31 21:55:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 5.914006116s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X................X.............................X...] duration:
561.201239ms
FAIL (2025-10-31 21:55:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 561.213952ms
edge
gateway
us-select-1
|