Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [X....X......................................................] duration:
455.434617ms
FAIL (2025-10-31 22:10:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 455.450797ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X.............................X......................] duration:
404.25882ms
FAIL (2025-10-31 22:10:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 404.26903ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X.................................X......................] duration:
328.107419ms
FAIL (2025-10-31 22:10:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 328.125914ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....X.X...............................X.....................] duration:
639.283451ms
FAIL (2025-10-31 22:10:18 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 639.488076ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [......X.................................X...................] duration:
531.87449ms
FAIL (2025-10-31 22:09:58 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 531.891081ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X.................................X.....................] duration:
360.615301ms
FAIL (2025-10-31 22:09:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 360.635067ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X....X.....................................................] duration:
467.285158ms
FAIL (2025-10-31 22:09:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 467.30802ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....X.X...............................X....................] duration:
611.183469ms
FAIL (2025-10-31 22:09:41 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 611.205892ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X.................................X....................] duration:
370.546191ms
FAIL (2025-10-31 22:09:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 370.584735ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X.............................X.....................] duration:
363.49525ms
FAIL (2025-10-31 22:09:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 363.51602ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [X.X....X....................................................] duration:
398.660903ms
FAIL (2025-10-31 22:09:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 398.698773ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X.............................X....................] duration:
359.990369ms
FAIL (2025-10-31 22:08:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 360.007702ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [.......X.................................X..................] duration:
538.071477ms
FAIL (2025-10-31 22:08:49 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 538.089221ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......X.X...............................X...................] duration:
591.478735ms
FAIL (2025-10-31 22:08:43 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 591.495656ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X.................................X...................] duration:
326.038893ms
FAIL (2025-10-31 22:08:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 326.058228ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X.X....X...................................................] duration:
379.30092ms
FAIL (2025-10-31 22:08:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 379.32784ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X.X....X..................................................] duration:
381.0683ms
FAIL (2025-10-31 22:08:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 381.091453ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [X.......X.................................X.................] duration:
510.81701ms
FAIL (2025-10-31 22:07:56 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 510.835485ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X.............................X...................] duration:
367.897555ms
FAIL (2025-10-31 22:07:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 367.910591ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X.................................X..................] duration:
377.094442ms
FAIL (2025-10-31 22:07:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 377.117144ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......X.X...............................X..................] duration:
651.950997ms
FAIL (2025-10-31 22:07:31 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 651.967066ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.X.......X.................................X................] duration:
587.958634ms
FAIL (2025-10-31 22:07:15 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 587.974083ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X.............................X..................] duration:
366.280714ms
FAIL (2025-10-31 22:07:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 366.291964ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X.X....X.................................................] duration:
431.295891ms
FAIL (2025-10-31 22:06:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 431.314847ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X.................................X.................] duration:
360.832297ms
FAIL (2025-10-31 22:06:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 360.855931ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X.X....X................................................] duration:
399.431419ms
FAIL (2025-10-31 22:06:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 399.448691ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [X.......X.X...............................X.................] duration:
644.057775ms
FAIL (2025-10-31 22:06:31 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 644.071982ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X.............................X.................] duration:
616.702948ms
FAIL (2025-10-31 22:06:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 616.718487ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [..X.......X.................................X...............] duration:
514.6249ms
FAIL (2025-10-31 22:06:01 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 514.645207ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X.X....X...............................................] duration:
369.792007ms
FAIL (2025-10-31 22:05:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 369.813247ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.X.......X.X...............................X................] duration:
680.021275ms
FAIL (2025-10-31 22:05:47 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 680.037214ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X........X.................................X................] duration:
391.722913ms
FAIL (2025-10-31 22:05:40 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 391.744934ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [...X.......X.................................X..............] duration:
573.499694ms
FAIL (2025-10-31 22:05:15 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 573.512638ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X.............................X................] duration:
381.200101ms
FAIL (2025-10-31 22:05:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 381.222524ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X.X....X..............................................] duration:
369.11572ms
FAIL (2025-10-31 22:05:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 369.135036ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..X.......X.X...............................X...............] duration:
611.853253ms
FAIL (2025-10-31 22:04:55 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 611.876216ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X.X....X.............................................] duration:
417.314361ms
FAIL (2025-10-31 22:04:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 417.337915ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X........X.................................X...............] duration:
439.340037ms
FAIL (2025-10-31 22:04:28 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 439.35812ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............X.............................X...............] duration:
424.074447ms
FAIL (2025-10-31 22:04:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 424.091189ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...X.......X.X...............................X..............] duration:
695.681149ms
FAIL (2025-10-31 22:04:13 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 695.698581ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [....X.......X.................................X.............] duration:
540.339329ms
FAIL (2025-10-31 22:04:00 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 540.359046ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X.X....X............................................] duration:
381.319073ms
FAIL (2025-10-31 22:03:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 381.336576ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.....X.......X.................................X............] duration:
542.962309ms
FAIL (2025-10-31 22:03:33 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 543.10667ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [....X.......X.X...............................X.............] duration:
594.461158ms
FAIL (2025-10-31 22:03:28 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 594.47789ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X........X.................................X..............] duration:
447.747641ms
FAIL (2025-10-31 22:03:20 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 447.773669ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............X.............................X..............] duration:
394.822786ms
FAIL (2025-10-31 22:03:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 394.886666ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [......X.......X.................................X...........] duration:
558.170709ms
FAIL (2025-10-31 22:03:13 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 558.188301ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X.X....X...........................................] duration:
536.388914ms
FAIL (2025-10-31 22:02:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 536.415734ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.....X.......X.X...............................X............] duration:
695.946006ms
FAIL (2025-10-31 22:02:51 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 695.959051ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X........X.................................X.............] duration:
492.420987ms
FAIL (2025-10-31 22:02:43 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 492.441406ms
core
uplink
us-select-1
|