Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...................................X.......X.X..............] duration:
645.288134ms
FAIL (2025-10-31 21:38:26 UTC)
orbiter warning: unexpected EOF
Duration: 645.491105ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............X.....................X........X..............] duration:
348.367909ms
FAIL (2025-10-31 21:38:11 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 348.386833ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............X............X..........X.X....X..............] duration:
431.95164ms
FAIL (2025-10-31 21:38:05 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 431.975134ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.........................X...........X.......X..............] duration:
508.312521ms
FAIL (2025-10-31 21:37:54 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 508.322871ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X.............X................X............] duration:
404.966161ms
FAIL (2025-10-31 21:37:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 404.983113ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............X............X..........X.X....X.............] duration:
464.63549ms
FAIL (2025-10-31 21:37:48 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 464.665356ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....................................X.......X.X.............] duration:
610.968018ms
FAIL (2025-10-31 21:37:44 UTC)
orbiter warning: unexpected EOF
Duration: 610.982775ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............X.....................X........X.............] duration:
394.868024ms
FAIL (2025-10-31 21:37:34 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 394.891538ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X................X.............X................X...........] duration:
334.421864ms
FAIL (2025-10-31 21:37:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 334.440039ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....................................X.......X.X............] duration:
639.995536ms
FAIL (2025-10-31 21:37:04 UTC)
orbiter warning: unexpected EOF
Duration: 640.012668ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [..........................X...........X.......X.............] duration:
502.643634ms
FAIL (2025-10-31 21:36:56 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 502.661268ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [................X............X..........X.X....X............] duration:
527.605709ms
FAIL (2025-10-31 21:36:52 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 527.628663ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X.....................X........X............] duration:
447.682041ms
FAIL (2025-10-31 21:36:49 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 447.701137ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X................X.............X................X..........] duration:
490.652515ms
FAIL (2025-10-31 21:36:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 490.671401ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................X.....................X........X...........] duration:
468.109954ms
FAIL (2025-10-31 21:36:28 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 468.134641ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [...........................X...........X.......X............] duration:
541.399054ms
FAIL (2025-10-31 21:36:28 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 541.41267ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......................................X.......X.X...........] duration:
644.414608ms
FAIL (2025-10-31 21:36:15 UTC)
unexpected EOF
Duration: 644.434214ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................X.....................X........X..........] duration:
352.412923ms
FAIL (2025-10-31 21:36:11 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 352.431698ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......................................X.......X.X..........] duration:
806.021227ms
FAIL (2025-10-31 21:35:56 UTC)
unexpected EOF
Duration: 806.042537ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................X............X..........X.X....X...........] duration:
446.752577ms
FAIL (2025-10-31 21:35:52 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 446.775439ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X................X.............X................X.........] duration:
363.029972ms
FAIL (2025-10-31 21:35:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 363.050781ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................X............X..........X.X....X..........] duration:
455.985206ms
FAIL (2025-10-31 21:35:30 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 456.123465ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [............................X...........X.......X...........] duration:
517.63361ms
FAIL (2025-10-31 21:35:29 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 517.651204ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................X.....................X........X.........] duration:
391.493846ms
FAIL (2025-10-31 21:35:25 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 391.515467ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X................X.............X................X........] duration:
384.693925ms
FAIL (2025-10-31 21:35:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 384.732408ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [........................................X.......X.X.........] duration:
659.996007ms
FAIL (2025-10-31 21:35:00 UTC)
unexpected EOF
Duration: 660.020053ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.............................X...........X.......X..........] duration:
536.420028ms
FAIL (2025-10-31 21:34:49 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 536.440977ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................X............X..........X.X....X.........] duration:
409.385756ms
FAIL (2025-10-31 21:34:42 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 409.404371ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................X.....................X........X........] duration:
384.425484ms
FAIL (2025-10-31 21:34:23 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 384.443577ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........................................X.......X.X........] duration:
620.711031ms
FAIL (2025-10-31 21:34:21 UTC)
unexpected EOF
Duration: 620.724055ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [..............................X...........X.......X.........] duration:
569.851888ms
FAIL (2025-10-31 21:34:10 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 569.871715ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X................X.............X................X.......] duration:
457.697229ms
FAIL (2025-10-31 21:34:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 457.711736ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................X............X..........X.X....X........] duration:
447.212504ms
FAIL (2025-10-31 21:33:53 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 447.23697ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [...............................X...........X.......X........] duration:
502.597166ms
FAIL (2025-10-31 21:33:44 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 502.614468ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................X.....................X........X.......] duration:
336.984682ms
FAIL (2025-10-31 21:33:43 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 337.005181ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........................................X.......X.X.......] duration:
604.546125ms
FAIL (2025-10-31 21:33:28 UTC)
unexpected EOF
Duration: 604.564649ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [................................X...........X.......X.......] duration:
509.976558ms
FAIL (2025-10-31 21:33:15 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 509.993311ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X................X.............X................X......] duration:
383.076425ms
FAIL (2025-10-31 21:33:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 383.096453ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................................X.......X.X......] duration:
641.960033ms
FAIL (2025-10-31 21:33:01 UTC)
unexpected EOF
Duration: 641.979088ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................X.....................X........X......] duration:
331.637066ms
FAIL (2025-10-31 21:32:51 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 331.651744ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................X............X..........X.X....X.......] duration:
382.795046ms
FAIL (2025-10-31 21:32:42 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 382.814964ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................X............X..........X.X....X......] duration:
423.30609ms
FAIL (2025-10-31 21:32:23 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 423.32211ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.................................X...........X.......X......] duration:
573.945113ms
FAIL (2025-10-31 21:32:22 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 573.963487ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [X...........................................X.......X.X.....] duration:
665.011638ms
FAIL (2025-10-31 21:32:09 UTC)
unexpected EOF
Duration: 665.1778ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................X.....................X........X.....] duration:
348.847606ms
FAIL (2025-10-31 21:32:07 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 348.872062ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................X............X..........X.X....X.....] duration:
495.330689ms
FAIL (2025-10-31 21:32:07 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 495.349134ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X................X.............X................X.....] duration:
386.850391ms
FAIL (2025-10-31 21:31:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 386.875257ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [........................X............X..........X.X....X....] duration:
408.918589ms
FAIL (2025-10-31 21:31:49 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 408.93557ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=3) [X.................................X...........X.......X.....] duration:
567.541053ms
FAIL (2025-10-31 21:31:20 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 567.560299ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X................X.............X................X....] duration:
352.049812ms
FAIL (2025-10-31 21:31:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 352.066996ms
edge
gateway
us-select-1
|