Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X..........X.X....X.................................] duration:
16.422560195s
FAIL (2025-10-31 21:55:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 16.422586575s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [................X.......X.X...............................X.] duration:
24.638637764s
FAIL (2025-10-31 21:55:19 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 24.638659354s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........X................X.............................X..] duration:
798.430067ms
FAIL (2025-10-31 21:55:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 798.453651ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=3) [....X...........X.......X.................................X.] duration:
806.433573ms
FAIL (2025-10-31 21:55:07 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 806.608501ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............X........X.................................X.] duration:
396.817387ms
FAIL (2025-10-31 21:54:59 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 396.838266ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........X..........X.X....X................................] duration:
48.544597988s
FAIL (2025-10-31 21:54:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 48.544621242s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.................X.......X.X...............................X] duration:
1m0.000479041s
FAIL (2025-10-31 21:54:49 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000490622s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........X................X.............................X.] duration:
21.584954089s
FAIL (2025-10-31 21:54:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 21.584969327s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X........X.................................X] duration:
30.018924272s
FAIL (2025-10-31 21:54:25 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:53:55.636571581 (106h40m22.337516497s) [upload=93876]: Starting upload
Oct 31 21:53:55.636723827 (106h40m22.337668742s) [upload=93876]: Done starting upload
Oct 31 21:53:55.636846417 (106h40m22.337791352s) [upload=93876]: Uploading segments...
Oct 31 21:53:55.637224375 (106h40m22.33816931s) [upload=93876]: Got next segment. Inline: false
Oct 31 21:53:55.637325275 (106h40m22.3382702s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:53:55.637334242 (106h40m22.338279156s) [upload=93876]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:53:55.657334926 (106h40m22.358279841s) [seg_pos={0 0}, upload=93876]: Begin upload segment...
Oct 31 21:53:55.657376123 (106h40m22.358321039s) [seg_pos={0 0}, upload=93876]: Done begin upload segment.
Oct 31 21:53:55.657404807 (106h40m22.358349723s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.657436727 (106h40m22.358381642s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658171756 (106h40m22.35911667s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658246776 (106h40m22.359191691s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658568789 (106h40m22.359513705s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.658921311 (106h40m22.359866225s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659015637 (106h40m22.359960553s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659398385 (106h40m22.36034331s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659537967 (106h40m22.360482892s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.659905977 (106h40m22.360850893s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660336394 (106h40m22.36128131s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660367613 (106h40m22.361312528s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.660772091 (106h40m22.361717007s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661090197 (106h40m22.362035113s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661165078 (106h40m22.362109993s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.661754223 (106h40m22.362699138s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Uploading piece...
Oct 31 21:53:55.749091443 (106h40m22.450036338s) [upload=93876]: Next returned nil segment
Oct 31 21:53:55.749134874 (106h40m22.450079769s) [upload=93876]: Waiting for error group managing segments...
Oct 31 21:53:55.758087501 (106h40m22.459032385s) [noise=true , address=145.79.227.18:4005 , piece_id=HUC3W2CMGFTS6ZMSXYO4KKPWAIXCIKAWYPDIVONYGLUS6YVJUXHA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.758740425 (106h40m22.459685301s) [noise=true , address=145.79.227.17:4970 , piece_id=DBPOIR3D4LPW5JE6AXTI7IFOCM7PF2XKPGIUGC4X5RWWJOWH6R4Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759108115 (106h40m22.46005299s) [noise=true , address=145.79.227.11:4729 , piece_id=PLZHKJ7OR6NK6CT4R3FLZWPGF25UIBY2C2IWVJRCDYKW23WLJI2A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759663576 (106h40m22.460608451s) [noise=true , address=145.79.227.27:4690 , piece_id=SGUSU5KRFKILSJVY3K62GTJVJMNIFVPWYMNJWJNWC5VD3KHSF5GA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.759779894 (106h40m22.460724779s) [noise=true , address=145.79.227.25:4618 , piece_id=FCL3VVBGZBH64KLA4XFLBKHPIR5AE4H2IZFKKSTELTLXTO3LJMQQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760041835 (106h40m22.46098672s) [noise=true , address=145.79.227.32:4835 , piece_id=NXMRRTHQUAJTFD6ZYOFLXOJL3CNIG5FW5OVNNBQZ6ATOJGHLX5SQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760200122 (106h40m22.461145007s) [noise=true , address=145.79.227.15:4874 , piece_id=G5ASGJ4T5J6RGYJAFR7CMCMYGUG6AWZJTEZVO3KLKK76CNL6P75Q, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760415316 (106h40m22.461360201s) [noise=true , address=145.79.227.21:4475 , piece_id=MJATSGZ5ICSAPNAYEKZLEXK56AG6HTINRM2WMYE3CJEJKLQYT7XA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760602286 (106h40m22.461547171s) [noise=true , address=145.79.227.27:4691 , piece_id=GF45MMIGRCDYQSDTWZHYER5VOHVKCGHEXDDOBMMEDMMI24C335JA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.760776313 (106h40m22.461721198s) [noise=true , address=145.79.227.31:4799 , piece_id=Y5CZ4LLK2NBIDXPWDAWEQZBRKWDKIRPCU72T25HWSD5BQW53Z2RQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.761557137 (106h40m22.462502022s) [noise=true , address=145.79.227.21:4474 , piece_id=KFV4SPNC47UGX5X5JRSLWA4ZC5Q5VI7ROZ4YFO7JMVWCWETDWDXQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:53:55.776736230 (106h40m22.477681114s) [noise=true , address=145.79.227.10:4402 , piece_id=Z646NZ7OWMIO6J2HVNJKNNXOGDYPHG7GHUUMGM3W54UNXITOQQHQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: <nil>
Oct 31 21:54:25.625986871 (106h40m52.326931776s) [noise=true , address=45.91.116.14:4190 , piece_id=D5BI7QM7LCOK2RAEMFUANO56ZOGJVFQDJPZTKXYKVO7JROH3CGVA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 5.161.89.252:47216->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626009954 (106h40m52.326954839s) [noise=true , address=77.90.31.13:4626 , piece_id=5DCPIYCL5BB3DAANE3XGJJSSVNFUF5EV7TEIK5HJYU4G4LZRNBOQ, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 5.161.89.252:12918->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626144627 (106h40m52.327089522s) [noise=true , address=77.90.31.18:4805 , piece_id=Q4RMSFCWFI67O6GGYRB5SEDACQUH3XEUPNJQUVTC2EI65NQ5TH3A, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:46724->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626160907 (106h40m52.327105803s) [noise=true , address=77.90.31.16:4734 , piece_id=7D44KYJPUMILGVSDXUX6L36YZKQIBXGYWLMZXJZ74HNVB6QQRVSA, seg_pos={0 0}, upload=93876]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 5.161.89.252:16936->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:25.626188069 (106h40m52.327132953s) [seg_pos={0 0}, upload=93876]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:25.643440038 (106h40m52.344384934s) [upload=93876]: Done uploading segments...
Oct 31 21:54:25.643452923 (106h40m52.344397818s) [upload=93876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.018939691s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............X................X.............................X] duration:
30.000645399s
FAIL (2025-10-31 21:54:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000658213s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=3) [.....X...........X.......X.................................X] duration:
5.000752072s
FAIL (2025-10-31 21:54:02 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000769584s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X........X.................................] duration:
375.590375ms
FAIL (2025-10-31 21:53:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 375.613909ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........X..........X.X....X...............................] duration:
445.590194ms
FAIL (2025-10-31 21:53:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 445.603669ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [......X...........X.......X.................................] duration:
529.227264ms
FAIL (2025-10-31 21:52:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 529.258383ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X................X.............................] duration:
331.62925ms
FAIL (2025-10-31 21:52:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 331.652975ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................X.......X.X...............................] duration:
655.350382ms
FAIL (2025-10-31 21:52:41 UTC)
orbiter warning: unexpected EOF
Duration: 655.453767ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................X........X................................] duration:
479.062258ms
FAIL (2025-10-31 21:52:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 479.0843ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........X..........X.X....X..............................] duration:
892.414692ms
FAIL (2025-10-31 21:52:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 892.42951ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.......X...........X.......X................................] duration:
525.216783ms
FAIL (2025-10-31 21:52:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 525.23645ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X........X...............................] duration:
317.978985ms
FAIL (2025-10-31 21:51:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 318.002149ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...................X.......X.X..............................] duration:
634.416809ms
FAIL (2025-10-31 21:51:50 UTC)
orbiter warning: unexpected EOF
Duration: 634.432448ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X.............X................X............................] duration:
399.277613ms
FAIL (2025-10-31 21:51:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 399.471204ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [............X..........X.X....X.............................] duration:
450.978956ms
FAIL (2025-10-31 21:51:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 450.996771ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................X........X..............................] duration:
472.848536ms
FAIL (2025-10-31 21:51:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 472.865096ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....................X.......X.X.............................] duration:
611.740218ms
FAIL (2025-10-31 21:51:09 UTC)
orbiter warning: unexpected EOF
Duration: 611.762689ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [........X...........X.......X...............................] duration:
500.225582ms
FAIL (2025-10-31 21:51:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 500.262761ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [X............X..........X.X....X............................] duration:
503.682135ms
FAIL (2025-10-31 21:50:54 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 503.706881ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....................X.......X.X............................] duration:
612.941527ms
FAIL (2025-10-31 21:50:50 UTC)
orbiter warning: unexpected EOF
Duration: 612.95844ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X.............X................X...........................] duration:
328.755322ms
FAIL (2025-10-31 21:50:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 328.770691ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X............X..........X.X....X...........................] duration:
427.355251ms
FAIL (2025-10-31 21:50:23 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 427.379708ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................X........X.............................] duration:
807.048369ms
FAIL (2025-10-31 21:50:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 807.065091ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X.............X................X..........................] duration:
394.116883ms
FAIL (2025-10-31 21:50:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 394.127744ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.........X...........X.......X..............................] duration:
499.854136ms
FAIL (2025-10-31 21:49:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 499.871017ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......................X.......X.X...........................] duration:
603.43334ms
FAIL (2025-10-31 21:49:50 UTC)
orbiter warning: unexpected EOF
Duration: 603.449479ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X............X..........X.X....X..........................] duration:
348.397204ms
FAIL (2025-10-31 21:49:28 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 348.421369ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [..........X...........X.......X.............................] duration:
529.333281ms
FAIL (2025-10-31 21:49:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 529.354189ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......................X.......X.X..........................] duration:
644.675454ms
FAIL (2025-10-31 21:49:22 UTC)
orbiter warning: unexpected EOF
Duration: 644.691104ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X.....................X........X............................] duration:
368.635966ms
FAIL (2025-10-31 21:49:06 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 368.655573ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X.............X................X.........................] duration:
380.721845ms
FAIL (2025-10-31 21:49:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 380.74617ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [...........X...........X.......X............................] duration:
471.290981ms
FAIL (2025-10-31 21:49:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 471.455158ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [........................X.......X.X.........................] duration:
667.803786ms
FAIL (2025-10-31 21:48:58 UTC)
orbiter warning: unexpected EOF
Duration: 667.824625ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X.............X................X........................] duration:
386.233289ms
FAIL (2025-10-31 21:48:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 386.249881ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X............X..........X.X....X.........................] duration:
386.59518ms
FAIL (2025-10-31 21:48:21 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 386.620086ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [............X...........X.......X...........................] duration:
496.736309ms
FAIL (2025-10-31 21:48:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 496.755254ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X.....................X........X...........................] duration:
536.309514ms
FAIL (2025-10-31 21:48:17 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 536.330694ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X.............X................X.......................] duration:
398.878592ms
FAIL (2025-10-31 21:48:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 399.004448ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........................X.......X.X........................] duration:
609.297938ms
FAIL (2025-10-31 21:48:13 UTC)
orbiter warning: unexpected EOF
Duration: 609.322365ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X............X..........X.X....X........................] duration:
592.693062ms
FAIL (2025-10-31 21:47:54 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 592.715223ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.............X...........X.......X..........................] duration:
554.08176ms
FAIL (2025-10-31 21:47:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 554.131483ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X.....................X........X..........................] duration:
595.726139ms
FAIL (2025-10-31 21:47:23 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 595.746827ms
core
uplink
us-select-1
|