Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X.............X................X......................] duration:
387.621785ms
FAIL (2025-10-31 21:47:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 387.642954ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........................X.......X.X.......................] duration:
685.447999ms
FAIL (2025-10-31 21:47:05 UTC)
orbiter warning: unexpected EOF
Duration: 685.467545ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X.....................X........X.........................] duration:
447.542869ms
FAIL (2025-10-31 21:46:51 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 447.571403ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................X.......X.X......................] duration:
620.465497ms
FAIL (2025-10-31 21:46:48 UTC)
orbiter warning: unexpected EOF
Duration: 620.507897ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X............X..........X.X....X.......................] duration:
455.883555ms
FAIL (2025-10-31 21:46:45 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 455.903833ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [..............X...........X.......X.........................] duration:
575.64372ms
FAIL (2025-10-31 21:46:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 575.661854ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X.............X................X.....................] duration:
404.937784ms
FAIL (2025-10-31 21:46:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 405.220254ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X.....................X........X........................] duration:
386.090583ms
FAIL (2025-10-31 21:46:03 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 386.127312ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [...............X...........X.......X........................] duration:
733.384748ms
FAIL (2025-10-31 21:45:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 733.402572ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............................X.......X.X.....................] duration:
664.617189ms
FAIL (2025-10-31 21:45:36 UTC)
orbiter warning: unexpected EOF
Duration: 664.637026ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X............X..........X.X....X......................] duration:
415.899778ms
FAIL (2025-10-31 21:45:33 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 415.925786ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X.............X................X....................] duration:
1.527711814s
FAIL (2025-10-31 21:45:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.527731311s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X.....................X........X.......................] duration:
365.517267ms
FAIL (2025-10-31 21:44:49 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 365.531263ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [................X...........X.......X.......................] duration:
800.575574ms
FAIL (2025-10-31 21:44:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 800.591494ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............................X.......X.X....................] duration:
670.464995ms
FAIL (2025-10-31 21:44:43 UTC)
orbiter warning: unexpected EOF
Duration: 670.483882ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X.....................X........X......................] duration:
419.932924ms
FAIL (2025-10-31 21:44:29 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 419.954444ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X............X..........X.X....X.....................] duration:
419.297152ms
FAIL (2025-10-31 21:44:24 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 419.313002ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X.............X................X...................] duration:
355.659423ms
FAIL (2025-10-31 21:44:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 355.679451ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X.....................X........X.....................] duration:
381.359473ms
FAIL (2025-10-31 21:44:09 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 381.37357ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.................X...........X.......X......................] duration:
524.756227ms
FAIL (2025-10-31 21:43:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 524.803685ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [........X............X..........X.X....X....................] duration:
443.713228ms
FAIL (2025-10-31 21:43:43 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 443.738396ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............................X.......X.X...................] duration:
624.97054ms
FAIL (2025-10-31 21:43:32 UTC)
orbiter warning: unexpected EOF
Duration: 625.00836ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X.....................X........X....................] duration:
428.5744ms
FAIL (2025-10-31 21:43:10 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 428.594637ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........X.............X................X..................] duration:
377.913823ms
FAIL (2025-10-31 21:43:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 377.943158ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........X............X..........X.X....X...................] duration:
396.897811ms
FAIL (2025-10-31 21:42:59 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 396.916977ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [..................X...........X.......X.....................] duration:
562.897026ms
FAIL (2025-10-31 21:42:59 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 562.916132ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............................X.......X.X..................] duration:
671.874008ms
FAIL (2025-10-31 21:42:41 UTC)
orbiter warning: unexpected EOF
Duration: 671.952414ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [...................X...........X.......X....................] duration:
543.79565ms
FAIL (2025-10-31 21:42:29 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 543.81169ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X.....................X........X...................] duration:
384.05645ms
FAIL (2025-10-31 21:42:08 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 384.074965ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........X.............X................X.................] duration:
351.1426ms
FAIL (2025-10-31 21:42:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 351.1641ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........X............X..........X.X....X..................] duration:
481.557883ms
FAIL (2025-10-31 21:41:50 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 481.580416ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [................................X.......X.X.................] duration:
638.735695ms
FAIL (2025-10-31 21:41:37 UTC)
orbiter warning: unexpected EOF
Duration: 638.768266ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............X.............X................X................] duration:
467.030121ms
FAIL (2025-10-31 21:41:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 467.043526ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [....................X...........X.......X...................] duration:
530.089497ms
FAIL (2025-10-31 21:41:19 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 530.107861ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........X.....................X........X..................] duration:
422.731222ms
FAIL (2025-10-31 21:41:00 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 422.753375ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........X............X..........X.X....X.................] duration:
407.424459ms
FAIL (2025-10-31 21:40:59 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 407.444716ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........X.....................X........X.................] duration:
394.401471ms
FAIL (2025-10-31 21:40:42 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 394.425765ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.....................X...........X.......X..................] duration:
502.896075ms
FAIL (2025-10-31 21:40:42 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 502.913798ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............X.............X................X...............] duration:
369.587859ms
FAIL (2025-10-31 21:40:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 369.606545ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................................X.......X.X................] duration:
620.727418ms
FAIL (2025-10-31 21:40:23 UTC)
orbiter warning: unexpected EOF
Duration: 620.745732ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............X.....................X........X................] duration:
390.983379ms
FAIL (2025-10-31 21:40:06 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 391.011762ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [............X............X..........X.X....X................] duration:
406.508411ms
FAIL (2025-10-31 21:39:47 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 406.536796ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [......................X...........X.......X.................] duration:
564.892939ms
FAIL (2025-10-31 21:39:46 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 564.912827ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............X.............X................X..............] duration:
336.311891ms
FAIL (2025-10-31 21:39:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 336.496228ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................................X.......X.X...............] duration:
641.357272ms
FAIL (2025-10-31 21:39:18 UTC)
orbiter warning: unexpected EOF
Duration: 641.389975ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.......................X...........X.......X................] duration:
502.831854ms
FAIL (2025-10-31 21:39:15 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 502.849748ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............X.....................X........X...............] duration:
379.846146ms
FAIL (2025-10-31 21:39:06 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 379.871283ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............X............X..........X.X....X...............] duration:
632.517878ms
FAIL (2025-10-31 21:38:58 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 632.545509ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [........................X...........X.......X...............] duration:
568.303624ms
FAIL (2025-10-31 21:38:43 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 568.340622ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............X.............X................X.............] duration:
361.152143ms
FAIL (2025-10-31 21:38:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 361.165398ms
edge
gateway
us-select-1
|