Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=3) [.X.................................X...........X.......X....] duration:
525.000644ms
FAIL (2025-10-31 21:31:03 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 525.015983ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................X.....................X........X....] duration:
478.024006ms
FAIL (2025-10-31 21:31:01 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 478.051538ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.X...........................................X.......X.X....] duration:
643.458754ms
FAIL (2025-10-31 21:30:58 UTC)
unexpected EOF
Duration: 643.471979ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X................X.............X................X...] duration:
403.18036ms
FAIL (2025-10-31 21:30:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 403.245913ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..X...........................................X.......X.X...] duration:
880.952956ms
FAIL (2025-10-31 21:30:37 UTC)
unexpected EOF
Duration: 880.969428ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........................X............X..........X.X....X...] duration:
428.73444ms
FAIL (2025-10-31 21:30:36 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 428.754337ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................X.....................X........X...] duration:
591.508199ms
FAIL (2025-10-31 21:30:24 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 591.528367ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=3) [..X.................................X...........X.......X...] duration:
605.251488ms
FAIL (2025-10-31 21:30:21 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 605.275272ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...X...........................................X.......X.X..] duration:
21.319358288s
FAIL (2025-10-31 21:29:57 UTC)
unexpected EOF
Duration: 21.319372013s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........................X............X..........X.X....X..] duration:
15.630282113s
FAIL (2025-10-31 21:29:48 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 15.630300857s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........X................X.............X................X..] duration:
1.254288012s
FAIL (2025-10-31 21:29:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.254304342s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [....X...........................................X.......X.X.] duration:
10.041679014s
FAIL (2025-10-31 21:29:31 UTC)
unexpected EOF
Duration: 10.041690444s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................X.....................X........X..] duration:
565.946085ms
FAIL (2025-10-31 21:29:16 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 565.967234ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.....X...........................................X.......X.X] duration:
32.231688778s
FAIL (2025-10-31 21:29:13 UTC)
unexpected EOF
Duration: 32.231724745s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=3) [...X.................................X...........X.......X..] duration:
663.144078ms
FAIL (2025-10-31 21:29:10 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 663.159305ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........X................X.............X................X.] duration:
388.39793ms
FAIL (2025-10-31 21:28:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 388.423489ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........................X............X..........X.X....X.] duration:
5.222425781s
FAIL (2025-10-31 21:28:29 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 5.222442954s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [............................X............X..........X.X....X] duration:
1m0.018256034s
FAIL (2025-10-31 21:28:24 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
CVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109647425 (106h13m50.81059232s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109869402 (106h13m50.810814287s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.109978557 (106h13m50.810923441s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110164685 (106h13m50.81110958s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110399856 (106h13m50.811344751s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110450070 (106h13m50.811394965s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110726599 (106h13m50.811671484s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.110789657 (106h13m50.811734543s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111003198 (106h13m50.811948093s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111150555 (106h13m50.812095439s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111291028 (106h13m50.812235923s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111562457 (106h13m50.812507342s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.111573357 (106h13m50.812518252s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Uploading piece...
Oct 31 21:27:24.185273470 (106h13m50.886218365s) [upload=93483]: Next returned nil segment
Oct 31 21:27:24.185306943 (106h13m50.886251838s) [upload=93483]: Waiting for error group managing segments...
Oct 31 21:27:24.198842094 (106h13m50.899786989s) [noise=true , address=145.79.227.11:4752 , piece_id=DK2ALNHAYCSZYC7GNSPADHOUYMX2G6SI7VOEE564XBUONMXESSEA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202566047 (106h13m50.903510942s) [noise=true , address=145.79.227.7:4326 , piece_id=UWKMPOGPLOEQ7RIN6HB4CPL6I2UBLKYZ4IABXKX35WDPO2LRAOMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.202751024 (106h13m50.903695909s) [noise=true , address=145.79.227.49:4368 , piece_id=5KFAGAIUHS45PRRPHM53IWSCVORDO4VKXA7QG2ULG4ZHHSDZN4XA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.206213055 (106h13m50.90715795s) [noise=true , address=145.79.227.33:4898 , piece_id=XFD2TVHIKDC3NGPFNY4VLO4EDWE2VTSB3RT32HINFPSCAVEVZQRA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.224640360 (106h13m50.925585246s) [noise=true , address=145.79.227.21:4482 , piece_id=T246K4A3QSUUWBUL2AECHGA2SYJKEIVSMGZ2E7HPEL7FTN5N666A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.225634635 (106h13m50.92657953s) [noise=true , address=145.79.227.35:4501 , piece_id=HPFY3UD3P3UKMH3F352UTYVE4VYJQHJEZE4O6KMN4DA6H2H7VEMA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:27:24.232531017 (106h13m50.933475902s) [noise=true , address=145.79.227.44:4223 , piece_id=WSYN5Z3UPYBLQ72OTGVCL4BI5PJA6ZUIHMFWMTPZNYMOUVX3OSNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:13.947408711 (106h14m40.648353626s) [noise=true , address=45.91.116.7:4038 , piece_id=HBDQD2VYRPK5SLBDT54E3BEG3SUMFVKY66JVD6626PR5TBOX7RUQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:21.276856063 (106h14m47.977800958s) [noise=true , address=45.91.116.11:4137 , piece_id=UQEYQVYU5NQLEMQJV2U33DEOOCYYO6DNX4NV2ZSK2E5U3OEIWWGA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: <nil>
Oct 31 21:28:24.069871428 (106h14m50.770816313s) [noise=true , address=77.90.31.15:4726 , piece_id=FJH2K5DUC6F6QZTU7BPDQTXKHBAFQ72SOCCAKCP6TCQO6G4FZUHQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: expected piece hash; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; write tcp 5.161.89.252:52754->77.90.31.15:4726: use of closed network connection; EOF
Oct 31 21:28:24.069950206 (106h14m50.770895091s) [noise=true , address=77.90.31.11:4568 , piece_id=UEZNOBJV3B2UERQSCBUBNEIXDYR6Y73ENA3GYKXH4PYZJNIQNFHA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: read tcp 5.161.89.252:58260->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.069974752 (106h14m50.770919647s) [noise=true , address=45.91.116.4:4947 , piece_id=BF5KBPTGYUL5TYX5BXUNGGS22FLVFNIVNVIQNRMKJIJICBYW64LA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31236->45.91.116.4:4947: use of closed network connection
Oct 31 21:28:24.070022812 (106h14m50.770967707s) [noise=true , address=45.91.116.8:4076 , piece_id=DEIGDLGK7BBKHJRAQOCLOLKXHZLIDV7BB2K7DN6DYI2U7Q2C33IA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1WWDBuhCg17uYobyN1WCxT6TdHXjHg9NknseutbiCpKdGo1Nd1, address:45.91.116.8:4076): protocol: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:29812->45.91.116.8:4076: use of closed network connection
Oct 31 21:28:24.070054221 (106h14m50.770999106s) [noise=true , address=77.90.31.13:4648 , piece_id=VMWDIWLJHK5KYGZY3EO6IE3DVN3W7VVAV5VAFDGREOG76QR5AWLA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15230->77.90.31.13:4648: use of closed network connection
Oct 31 21:28:24.070113412 (106h14m50.771058297s) [noise=true , address=77.90.31.10:4546 , piece_id=OZQE54VEKSLC6AWZVREEH4ES6XFLBE6DI5C7QAP2Y6264AGXTPNQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:24.070129752 (106h14m50.771074638s) [noise=true , address=77.90.31.12:4621 , piece_id=GT2ZFVCXDEMMP2ZXGHCDOY2IGNKTW47TV26WJOACMKPNN6OY5TXA, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: read tcp 5.161.89.252:61442->77.90.31.12:4621: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:24.070184245 (106h14m50.77112914s) [noise=true , address=77.90.31.16:4737 , piece_id=R7454NFIMETFWXIWUJZAXRYVSZSM25UIQITZA6GNXDOXNZXDNY7A, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:16260->77.90.31.16:4737: use of closed network connection
Oct 31 21:28:24.070274645 (106h14m50.77121954s) [noise=true , address=45.91.116.8:4054 , piece_id=ZDA72BIAED6RS2QWNVFNOQWEDXFX4USJ6HCELZ2ZS6DOOG5WBQWQ, seg_pos={0 0}, upload=93483]: Done uploading piece. err: ecclient: upload failed (node:1qC4ZTAobMwMUu7DQ34re4G4TV1jGEHg2UBXbaEM4k12EggNJW, address:45.91.116.8:4054): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:24.070293379 (106h14m50.771238265s) [seg_pos={0 0}, upload=93483]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Oct 31 21:28:24.087202136 (106h14m50.788147031s) [upload=93483]: Done uploading segments...
Oct 31 21:28:24.087224858 (106h14m50.788169744s) [upload=93483]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.018272735s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=3) [....X.................................X...........X.......X.] duration:
929.807917ms
FAIL (2025-10-31 21:28:23 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 929.824948ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......X...........................................X.......X.] duration:
10.547899329s
FAIL (2025-10-31 21:28:17 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.547910861s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................X.....................X........X.] duration:
9.466957658s
FAIL (2025-10-31 21:28:13 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 9.466973889s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......X...........................................X.......X] duration:
1m0.00083734s
FAIL (2025-10-31 21:28:07 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000854404s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........X................X.............X................X] duration:
30.001379919s
FAIL (2025-10-31 21:27:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001402392s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................X.....................X........X] duration:
30.019924207s
FAIL (2025-10-31 21:27:42 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:12.873017784 (106h13m39.57396267s) [upload=93479]: Starting upload
Oct 31 21:27:12.873069291 (106h13m39.574014186s) [upload=93479]: Done starting upload
Oct 31 21:27:12.873138431 (106h13m39.574083316s) [upload=93479]: Uploading segments...
Oct 31 21:27:12.873304112 (106h13m39.574248997s) [upload=93479]: Got next segment. Inline: false
Oct 31 21:27:12.873339177 (106h13m39.574284073s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:12.873344748 (106h13m39.574289643s) [upload=93479]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:12.894469702 (106h13m39.595414587s) [seg_pos={0 0}, upload=93479]: Begin upload segment...
Oct 31 21:27:12.894523563 (106h13m39.595468448s) [seg_pos={0 0}, upload=93479]: Done begin upload segment.
Oct 31 21:27:12.894548610 (106h13m39.595493495s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894554151 (106h13m39.595499035s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894631265 (106h13m39.59557615s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.894943831 (106h13m39.595888716s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895253721 (106h13m39.596198606s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895587507 (106h13m39.596532392s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.895965436 (106h13m39.596910321s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896328207 (106h13m39.597273102s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.896674015 (106h13m39.5976189s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897068876 (106h13m39.598013771s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.897458807 (106h13m39.598403692s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898475663 (106h13m39.599420558s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.898610156 (106h13m39.59955504s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.899470339 (106h13m39.600415234s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.901232393 (106h13m39.602177277s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.902220496 (106h13m39.603165381s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Uploading piece...
Oct 31 21:27:12.979032072 (106h13m39.679976967s) [upload=93479]: Next returned nil segment
Oct 31 21:27:12.979043745 (106h13m39.67998864s) [upload=93479]: Waiting for error group managing segments...
Oct 31 21:27:12.989772322 (106h13m39.690717217s) [noise=true , address=145.79.227.18:4007 , piece_id=E46JXJ5SU7ZXWZNCOLFPVNY2HREWW6KYUY3TS44PZWRNNGHVGE2A, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990037339 (106h13m39.690982234s) [noise=true , address=145.79.227.4:4185 , piece_id=XD7CHIX6C6EGY54NYHLX24EGUEEIR3FOGGW7ELGKJZZVSD3W5XYQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990116708 (106h13m39.691061603s) [noise=true , address=145.79.227.32:4834 , piece_id=AEVSB7MHSFNVAIFHFEAZ64RE2ZD4Q5MAXWDZUVZAY4WJAL5IVXJQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990744245 (106h13m39.69168913s) [noise=true , address=145.79.227.28:4725 , piece_id=MCBLVNF6CDLX5UNJ6K4GPW2CIBAAK5W2VTI7R4IV2FF6W4ALL2WA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.990997460 (106h13m39.691942405s) [noise=true , address=145.79.227.14:4838 , piece_id=CDXSI2L235OBJZUPYJHDQ5OLHTG5FCGGMTGRHRPMHG6IIYBWDQAQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991112886 (106h13m39.692057771s) [noise=true , address=145.79.227.30:4763 , piece_id=PA5TP2MMIQFKF2Z6VZR23ZKDHDPQ6C5E7DOXV672UOKVLZOPE2BQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991524769 (106h13m39.692469654s) [noise=true , address=145.79.227.15:4875 , piece_id=2ZVNBKT5ZWCBRZOCO5ELNW7RAZVVXQIL3T6T6AUBBCUYCWDZKXXQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.991565084 (106h13m39.692509979s) [noise=true , address=145.79.227.31:4797 , piece_id=I2D7MVAD6BWDQT5VNZZ5TGFHVQJ7Y5AXSRWTT73EED3UN56MOZNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.992416622 (106h13m39.693361506s) [noise=true , address=145.79.227.13:4801 , piece_id=HOTDTCXVEDXOE3CCCDBGZRLJEFVQSRHJNCC6K2TYIHDVKWWFARJA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:12.994088576 (106h13m39.695033461s) [noise=true , address=145.79.227.10:4402 , piece_id=JDBSXUPLMERY3QHW2FGW7FOIT3FOAHAQIMX6ZVV5VZBUHAIH7HTA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.003222133 (106h13m39.704167028s) [noise=true , address=145.79.227.20:4438 , piece_id=VAV4S443OSYWUE3SCUOQEUDAVEBNS2CEW7NBCGUXTZYX55IV5L5Q, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:13.039202404 (106h13m39.740147329s) [noise=true , address=145.79.227.24:4583 , piece_id=XEAIDN7P2POVBBJ5OIAY7E7DMIXQ4HYOSIOR4CWPM5A3VFO74WFA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: <nil>
Oct 31 21:27:42.861708993 (106h14m9.562653878s) [noise=true , address=77.90.31.18:4806 , piece_id=F7OCYHQPABP755YXLW4LZD5S5OHUQ3V5YBFKMVGQDAD3D6FP76AQ, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:64524->77.90.31.18:4806: use of closed network connection
Oct 31 21:27:42.861739491 (106h14m9.562684375s) [noise=true , address=77.90.31.10:4520 , piece_id=2PL7IDKYOLPSIJCJH6V6BYE45ZGBBONC6UDKS2E4W3HYD62XGCOA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17996->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:42.861846311 (106h14m9.562791196s) [noise=true , address=77.90.31.14:4663 , piece_id=T4OAOL3VLYADXDAYSUWFFKBLAEFAZWRVFOX2AOWXTYGDGLW2DGNA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 5.161.89.252:9944->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861907145 (106h14m9.56285202s) [noise=true , address=77.90.31.12:4589 , piece_id=YYX6E2KIUSLII3YQLWZTJMC54ZYACD3LXF7HOXNHPQE72NSCP2PA, seg_pos={0 0}, upload=93479]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 5.161.89.252:31980->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:42.861931170 (106h14m9.562876055s) [seg_pos={0 0}, upload=93479]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:42.879961209 (106h14m9.580906094s) [upload=93479]: Done uploading segments...
Oct 31 21:27:42.879980225 (106h14m9.58092512s) [upload=93479]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.019940597s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=3) [.....X.................................X...........X.......X] duration:
5.000650185s
FAIL (2025-10-31 21:27:38 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000667608s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............................X............X..........X.X....] duration:
435.959517ms
FAIL (2025-10-31 21:26:51 UTC)
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
K3VNVMW4D5ZS4QLA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.936453020 (106h9m38.637397935s) [noise=true , address=45.91.116.4:4943 , piece_id=243RKZW4BEML4R2FFSYQJPAMWH4CNEPLOA6HNF2QE34KZRDPNXMQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.936945263 (106h9m38.637890188s) [noise=true , address=77.90.31.12:4599 , piece_id=U74XV7GO2KBUCAAS7TM2X5B7HPRU65Z7OFVBAHH3FSEM3RD4UEWQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.937417759 (106h9m38.638362674s) [noise=true , address=45.91.116.7:4034 , piece_id=W5M5KDA5ZUBO6NH3ZDA6DDGZJVCRNOSC5GQO6ES6T52DIXGFRHQQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.937844629 (106h9m38.638789544s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.938276769 (106h9m38.639221684s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.938759274 (106h9m38.6397042s) [noise=true , address=45.91.116.4:4963 , piece_id=B6QJWTQOHC7V7EEWZ2MLXH7DXMGGDK5AMX6WNNGO5TVKJJTEWMVA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.939226250 (106h9m38.640171165s) [noise=true , address=45.91.116.5:4326 , piece_id=3DCJH36YX5NCWBCI7WPI6XRRCIAZLF4L64BAMUHCOJL5LDXHST2Q, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.939663400 (106h9m38.640608315s) [noise=true , address=145.79.227.38:4998 , piece_id=QOYPHU6BDTLOMIA2MOBQVA6DEKTRKTJS4XEMTUWYFJ2GCWTBCFGQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.940657263 (106h9m38.641602178s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.941228184 (106h9m38.642173098s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.946730872 (106h9m38.647675787s) [noise=true , address=77.90.31.16:4759 , piece_id=QHJCJ5YSBE6CPMVIQBNOMGGVIGHJ2OWWVMOLPL5SQVJ4UFZI24PQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.948817905 (106h9m38.649762821s) [noise=true , address=45.91.116.13:4160 , piece_id=JGTRITASZGTS2FEJBR5M2LDDJ36PKYR3EFJMAH6CJ4JTMTZUZE2Q, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.950060556 (106h9m38.651005471s) [noise=true , address=45.91.116.10:4085 , piece_id=FGQTDJBIGMXW2F7FHJSJ7H5RYCKGCEIPE44E3DI6CJTQZWKOEVRA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.956241747 (106h9m38.657186672s) [noise=true , address=145.79.227.43:4163 , piece_id=OXKUQ6IF564LJIUTUKSBL6XEJ23GUTE5ALJCBLLRN42M5YNWIUCQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.958515621 (106h9m38.659460537s) [noise=true , address=145.79.227.38:4006 , piece_id=F2WDTLGHA4RWUTMZQIOABFHNFULMX2D355MEZJNFMUU57ATGWACQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.961843962 (106h9m38.662788887s) [noise=true , address=145.79.227.13:4836 , piece_id=K2XFNVC3RXKXEBF4GTNU56B5H6YTEBOCKVBKQV2PSVYMBME5A2XQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:12.063468287 (106h9m38.764413202s) [upload=93420]: Next returned nil segment
Oct 31 21:23:12.063488516 (106h9m38.76443343s) [upload=93420]: Waiting for error group managing segments...
Oct 31 21:23:12.075808427 (106h9m38.776753342s) [noise=true , address=145.79.227.43:4163 , piece_id=OXKUQ6IF564LJIUTUKSBL6XEJ23GUTE5ALJCBLLRN42M5YNWIUCQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.105232644 (106h9m38.806177549s) [noise=true , address=145.79.227.38:4998 , piece_id=QOYPHU6BDTLOMIA2MOBQVA6DEKTRKTJS4XEMTUWYFJ2GCWTBCFGQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.107710410 (106h9m38.808655325s) [noise=true , address=145.79.227.13:4836 , piece_id=K2XFNVC3RXKXEBF4GTNU56B5H6YTEBOCKVBKQV2PSVYMBME5A2XQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.118028529 (106h9m38.818973444s) [noise=true , address=45.91.116.4:4943 , piece_id=243RKZW4BEML4R2FFSYQJPAMWH4CNEPLOA6HNF2QE34KZRDPNXMQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.137402949 (106h9m38.838347955s) [noise=true , address=45.91.116.7:4034 , piece_id=W5M5KDA5ZUBO6NH3ZDA6DDGZJVCRNOSC5GQO6ES6T52DIXGFRHQQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.278017809 (106h9m38.978962695s) [noise=true , address=45.91.116.5:4326 , piece_id=3DCJH36YX5NCWBCI7WPI6XRRCIAZLF4L64BAMUHCOJL5LDXHST2Q, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.333784943 (106h9m39.034729828s) [noise=true , address=45.91.116.10:4085 , piece_id=FGQTDJBIGMXW2F7FHJSJ7H5RYCKGCEIPE44E3DI6CJTQZWKOEVRA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.367267139 (106h9m39.068212034s) [noise=true , address=45.91.116.8:4072 , piece_id=77IRSHYY2UV2GKIV2GLS63G4SP2GKTCAYCT6K3VNVMW4D5ZS4QLA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.400124803 (106h9m39.101069718s) [noise=true , address=45.91.116.4:4963 , piece_id=B6QJWTQOHC7V7EEWZ2MLXH7DXMGGDK5AMX6WNNGO5TVKJJTEWMVA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.573123492 (106h9m39.274068387s) [noise=true , address=145.79.227.38:4006 , piece_id=F2WDTLGHA4RWUTMZQIOABFHNFULMX2D355MEZJNFMUU57ATGWACQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.617098341 (106h9m39.318043227s) [noise=true , address=45.91.116.13:4160 , piece_id=JGTRITASZGTS2FEJBR5M2LDDJ36PKYR3EFJMAH6CJ4JTMTZUZE2Q, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:13.345716310 (106h9m40.046661225s) [noise=true , address=77.90.31.12:4599 , piece_id=U74XV7GO2KBUCAAS7TM2X5B7HPRU65Z7OFVBAHH3FSEM3RD4UEWQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:13.827713328 (106h9m40.528658223s) [noise=true , address=77.90.31.16:4759 , piece_id=QHJCJ5YSBE6CPMVIQBNOMGGVIGHJ2OWWVMOLPL5SQVJ4UFZI24PQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:14.056029484 (106h9m40.75697437s) [noise=true , address=77.90.31.21:4911 , piece_id=BQVGAE7XLX3KVML7PAB5XCHU5PUQBQHCTUAJJ2D5HIQYAGWFWSRQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:14.056046707 (106h9m40.756991591s) [seg_pos={0 0}, upload=93420]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:23:14.056146393 (106h9m40.757091279s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF
Oct 31 21:23:14.056153627 (106h9m40.757098512s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056437710 (106h9m40.757382595s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k29MGpez3ByMHnG8qGBm9iPwo3VrKUGDHpz9CqGEGNn47rxDx): protocol: rpc: dial tcp 77.90.31.14:4679: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:23:14.056445925 (106h9m40.75739081s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056536505 (106h9m40.757481441s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF
Oct 31 21:23:14.056543578 (106h9m40.757488483s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056652873 (106h9m40.757597768s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; write tcp 5.161.89.252:34960->77.90.31.19:4862: use of closed network connection; write tcp 5.161.89.252:34960->77.90.31.19:4862: use of closed network connection; EOF
Oct 31 21:23:14.056660638 (106h9m40.757605523s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056676818 (106h9m40.757621693s) [seg_pos={0 0}, upload=93420]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:23:14.056740047 (106h9m40.757684932s) [upload=93420]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:23:14.056746409 (106h9m40.757691294s) [upload=93420]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:23:14.097207661 (106h9m40.798152547s) [upload=93420]: Done uploading segments...
Oct 31 21:23:14.097221608 (106h9m40.798166482s) [upload=93420]: Upload finished. err: <nil>
Duration: 435.980176ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X.....................X........] duration:
357.793963ms
FAIL (2025-10-31 21:26:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:22:00.944281574 (106h8m27.645226459s) [upload=93401]: Starting upload
Oct 31 21:22:00.944356304 (106h8m27.645301199s) [upload=93401]: Done starting upload
Oct 31 21:22:00.944468063 (106h8m27.645412948s) [upload=93401]: Uploading segments...
Oct 31 21:22:00.944630548 (106h8m27.645575433s) [upload=93401]: Got next segment. Inline: false
Oct 31 21:22:00.944665213 (106h8m27.645610108s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:00.944670022 (106h8m27.645614907s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:00.963897347 (106h8m27.664842243s) [seg_pos={0 0}, upload=93401]: Begin upload segment...
Oct 31 21:22:00.963998978 (106h8m27.664943873s) [seg_pos={0 0}, upload=93401]: Done begin upload segment.
Oct 31 21:22:00.964028473 (106h8m27.664973368s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964037450 (106h8m27.664982345s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964444233 (106h8m27.665389128s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964892844 (106h8m27.665837739s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965325144 (106h8m27.666270039s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965625197 (106h8m27.666570082s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966019857 (106h8m27.666964752s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966370855 (106h8m27.66731574s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966687539 (106h8m27.667632424s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.967022156 (106h8m27.667967042s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.968385533 (106h8m27.669330428s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.969631550 (106h8m27.670576435s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.971122787 (106h8m27.672067672s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.972500710 (106h8m27.673445595s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.973763789 (106h8m27.674708684s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.974994406 (106h8m27.675939291s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:01.045346730 (106h8m27.746291626s) [upload=93401]: Next returned nil segment
Oct 31 21:22:01.045360827 (106h8m27.746305712s) [upload=93401]: Waiting for error group managing segments...
Oct 31 21:22:01.055543281 (106h8m27.756488176s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055753104 (106h8m27.75669798s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055999196 (106h8m27.756944081s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056605764 (106h8m27.757550649s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056939789 (106h8m27.757884675s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057268406 (106h8m27.758213291s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057510931 (106h8m27.758455815s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057806124 (106h8m27.758751009s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.058699179 (106h8m27.759644064s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059036802 (106h8m27.759981688s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059484752 (106h8m27.760429636s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:30.929275627 (106h8m57.630220522s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929342072 (106h8m57.630286956s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:31798->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929405521 (106h8m57.630350416s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929460374 (106h8m57.630405269s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:53362->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929546896 (106h8m57.630491791s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 5.161.89.252:26862->77.90.31.19:4841: use of closed network connection; context deadline exceeded
Oct 31 21:22:30.929573285 (106h8m57.63051818s) [seg_pos={0 0}, upload=93401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:22:30.950873398 (106h8m57.651818293s) [upload=93401]: Done uploading segments...
Oct 31 21:22:30.950890520 (106h8m57.651835415s) [upload=93401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 357.81934ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [......X.................................X...........X.......] duration:
484.51508ms
FAIL (2025-10-31 21:26:33 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 484.53122ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........X...........................................X.......] duration:
644.711529ms
FAIL (2025-10-31 21:26:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 644.727739ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............X................X.............X................] duration:
375.443168ms
FAIL (2025-10-31 21:26:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 375.456884ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.......X.................................X...........X......] duration:
535.975503ms
FAIL (2025-10-31 21:26:10 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 536.023033ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X.....................X.......] duration:
375.192507ms
FAIL (2025-10-31 21:26:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:22:00.944281574 (106h8m27.645226459s) [upload=93401]: Starting upload
Oct 31 21:22:00.944356304 (106h8m27.645301199s) [upload=93401]: Done starting upload
Oct 31 21:22:00.944468063 (106h8m27.645412948s) [upload=93401]: Uploading segments...
Oct 31 21:22:00.944630548 (106h8m27.645575433s) [upload=93401]: Got next segment. Inline: false
Oct 31 21:22:00.944665213 (106h8m27.645610108s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:00.944670022 (106h8m27.645614907s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:00.963897347 (106h8m27.664842243s) [seg_pos={0 0}, upload=93401]: Begin upload segment...
Oct 31 21:22:00.963998978 (106h8m27.664943873s) [seg_pos={0 0}, upload=93401]: Done begin upload segment.
Oct 31 21:22:00.964028473 (106h8m27.664973368s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964037450 (106h8m27.664982345s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964444233 (106h8m27.665389128s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964892844 (106h8m27.665837739s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965325144 (106h8m27.666270039s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965625197 (106h8m27.666570082s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966019857 (106h8m27.666964752s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966370855 (106h8m27.66731574s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966687539 (106h8m27.667632424s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.967022156 (106h8m27.667967042s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.968385533 (106h8m27.669330428s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.969631550 (106h8m27.670576435s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.971122787 (106h8m27.672067672s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.972500710 (106h8m27.673445595s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.973763789 (106h8m27.674708684s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.974994406 (106h8m27.675939291s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:01.045346730 (106h8m27.746291626s) [upload=93401]: Next returned nil segment
Oct 31 21:22:01.045360827 (106h8m27.746305712s) [upload=93401]: Waiting for error group managing segments...
Oct 31 21:22:01.055543281 (106h8m27.756488176s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055753104 (106h8m27.75669798s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055999196 (106h8m27.756944081s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056605764 (106h8m27.757550649s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056939789 (106h8m27.757884675s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057268406 (106h8m27.758213291s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057510931 (106h8m27.758455815s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057806124 (106h8m27.758751009s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.058699179 (106h8m27.759644064s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059036802 (106h8m27.759981688s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059484752 (106h8m27.760429636s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:30.929275627 (106h8m57.630220522s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929342072 (106h8m57.630286956s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:31798->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929405521 (106h8m57.630350416s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929460374 (106h8m57.630405269s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:53362->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929546896 (106h8m57.630491791s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 5.161.89.252:26862->77.90.31.19:4841: use of closed network connection; context deadline exceeded
Oct 31 21:22:30.929573285 (106h8m57.63051818s) [seg_pos={0 0}, upload=93401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:22:30.950873398 (106h8m57.651818293s) [upload=93401]: Done uploading segments...
Oct 31 21:22:30.950890520 (106h8m57.651835415s) [upload=93401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 375.216562ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............X................X.............X...............] duration:
345.174848ms
FAIL (2025-10-31 21:25:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 345.191108ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............................X............X..........X.X...] duration:
370.061555ms
FAIL (2025-10-31 21:25:41 UTC)
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
K3VNVMW4D5ZS4QLA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.936453020 (106h9m38.637397935s) [noise=true , address=45.91.116.4:4943 , piece_id=243RKZW4BEML4R2FFSYQJPAMWH4CNEPLOA6HNF2QE34KZRDPNXMQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.936945263 (106h9m38.637890188s) [noise=true , address=77.90.31.12:4599 , piece_id=U74XV7GO2KBUCAAS7TM2X5B7HPRU65Z7OFVBAHH3FSEM3RD4UEWQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.937417759 (106h9m38.638362674s) [noise=true , address=45.91.116.7:4034 , piece_id=W5M5KDA5ZUBO6NH3ZDA6DDGZJVCRNOSC5GQO6ES6T52DIXGFRHQQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.937844629 (106h9m38.638789544s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.938276769 (106h9m38.639221684s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.938759274 (106h9m38.6397042s) [noise=true , address=45.91.116.4:4963 , piece_id=B6QJWTQOHC7V7EEWZ2MLXH7DXMGGDK5AMX6WNNGO5TVKJJTEWMVA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.939226250 (106h9m38.640171165s) [noise=true , address=45.91.116.5:4326 , piece_id=3DCJH36YX5NCWBCI7WPI6XRRCIAZLF4L64BAMUHCOJL5LDXHST2Q, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.939663400 (106h9m38.640608315s) [noise=true , address=145.79.227.38:4998 , piece_id=QOYPHU6BDTLOMIA2MOBQVA6DEKTRKTJS4XEMTUWYFJ2GCWTBCFGQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.940657263 (106h9m38.641602178s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.941228184 (106h9m38.642173098s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.946730872 (106h9m38.647675787s) [noise=true , address=77.90.31.16:4759 , piece_id=QHJCJ5YSBE6CPMVIQBNOMGGVIGHJ2OWWVMOLPL5SQVJ4UFZI24PQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.948817905 (106h9m38.649762821s) [noise=true , address=45.91.116.13:4160 , piece_id=JGTRITASZGTS2FEJBR5M2LDDJ36PKYR3EFJMAH6CJ4JTMTZUZE2Q, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.950060556 (106h9m38.651005471s) [noise=true , address=45.91.116.10:4085 , piece_id=FGQTDJBIGMXW2F7FHJSJ7H5RYCKGCEIPE44E3DI6CJTQZWKOEVRA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.956241747 (106h9m38.657186672s) [noise=true , address=145.79.227.43:4163 , piece_id=OXKUQ6IF564LJIUTUKSBL6XEJ23GUTE5ALJCBLLRN42M5YNWIUCQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.958515621 (106h9m38.659460537s) [noise=true , address=145.79.227.38:4006 , piece_id=F2WDTLGHA4RWUTMZQIOABFHNFULMX2D355MEZJNFMUU57ATGWACQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.961843962 (106h9m38.662788887s) [noise=true , address=145.79.227.13:4836 , piece_id=K2XFNVC3RXKXEBF4GTNU56B5H6YTEBOCKVBKQV2PSVYMBME5A2XQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:12.063468287 (106h9m38.764413202s) [upload=93420]: Next returned nil segment
Oct 31 21:23:12.063488516 (106h9m38.76443343s) [upload=93420]: Waiting for error group managing segments...
Oct 31 21:23:12.075808427 (106h9m38.776753342s) [noise=true , address=145.79.227.43:4163 , piece_id=OXKUQ6IF564LJIUTUKSBL6XEJ23GUTE5ALJCBLLRN42M5YNWIUCQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.105232644 (106h9m38.806177549s) [noise=true , address=145.79.227.38:4998 , piece_id=QOYPHU6BDTLOMIA2MOBQVA6DEKTRKTJS4XEMTUWYFJ2GCWTBCFGQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.107710410 (106h9m38.808655325s) [noise=true , address=145.79.227.13:4836 , piece_id=K2XFNVC3RXKXEBF4GTNU56B5H6YTEBOCKVBKQV2PSVYMBME5A2XQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.118028529 (106h9m38.818973444s) [noise=true , address=45.91.116.4:4943 , piece_id=243RKZW4BEML4R2FFSYQJPAMWH4CNEPLOA6HNF2QE34KZRDPNXMQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.137402949 (106h9m38.838347955s) [noise=true , address=45.91.116.7:4034 , piece_id=W5M5KDA5ZUBO6NH3ZDA6DDGZJVCRNOSC5GQO6ES6T52DIXGFRHQQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.278017809 (106h9m38.978962695s) [noise=true , address=45.91.116.5:4326 , piece_id=3DCJH36YX5NCWBCI7WPI6XRRCIAZLF4L64BAMUHCOJL5LDXHST2Q, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.333784943 (106h9m39.034729828s) [noise=true , address=45.91.116.10:4085 , piece_id=FGQTDJBIGMXW2F7FHJSJ7H5RYCKGCEIPE44E3DI6CJTQZWKOEVRA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.367267139 (106h9m39.068212034s) [noise=true , address=45.91.116.8:4072 , piece_id=77IRSHYY2UV2GKIV2GLS63G4SP2GKTCAYCT6K3VNVMW4D5ZS4QLA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.400124803 (106h9m39.101069718s) [noise=true , address=45.91.116.4:4963 , piece_id=B6QJWTQOHC7V7EEWZ2MLXH7DXMGGDK5AMX6WNNGO5TVKJJTEWMVA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.573123492 (106h9m39.274068387s) [noise=true , address=145.79.227.38:4006 , piece_id=F2WDTLGHA4RWUTMZQIOABFHNFULMX2D355MEZJNFMUU57ATGWACQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.617098341 (106h9m39.318043227s) [noise=true , address=45.91.116.13:4160 , piece_id=JGTRITASZGTS2FEJBR5M2LDDJ36PKYR3EFJMAH6CJ4JTMTZUZE2Q, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:13.345716310 (106h9m40.046661225s) [noise=true , address=77.90.31.12:4599 , piece_id=U74XV7GO2KBUCAAS7TM2X5B7HPRU65Z7OFVBAHH3FSEM3RD4UEWQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:13.827713328 (106h9m40.528658223s) [noise=true , address=77.90.31.16:4759 , piece_id=QHJCJ5YSBE6CPMVIQBNOMGGVIGHJ2OWWVMOLPL5SQVJ4UFZI24PQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:14.056029484 (106h9m40.75697437s) [noise=true , address=77.90.31.21:4911 , piece_id=BQVGAE7XLX3KVML7PAB5XCHU5PUQBQHCTUAJJ2D5HIQYAGWFWSRQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:14.056046707 (106h9m40.756991591s) [seg_pos={0 0}, upload=93420]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:23:14.056146393 (106h9m40.757091279s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF
Oct 31 21:23:14.056153627 (106h9m40.757098512s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056437710 (106h9m40.757382595s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k29MGpez3ByMHnG8qGBm9iPwo3VrKUGDHpz9CqGEGNn47rxDx): protocol: rpc: dial tcp 77.90.31.14:4679: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:23:14.056445925 (106h9m40.75739081s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056536505 (106h9m40.757481441s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF
Oct 31 21:23:14.056543578 (106h9m40.757488483s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056652873 (106h9m40.757597768s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; write tcp 5.161.89.252:34960->77.90.31.19:4862: use of closed network connection; write tcp 5.161.89.252:34960->77.90.31.19:4862: use of closed network connection; EOF
Oct 31 21:23:14.056660638 (106h9m40.757605523s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056676818 (106h9m40.757621693s) [seg_pos={0 0}, upload=93420]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:23:14.056740047 (106h9m40.757684932s) [upload=93420]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:23:14.056746409 (106h9m40.757691294s) [upload=93420]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:23:14.097207661 (106h9m40.798152547s) [upload=93420]: Done uploading segments...
Oct 31 21:23:14.097221608 (106h9m40.798166482s) [upload=93420]: Upload finished. err: <nil>
Duration: 370.081002ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........X...........................................X......] duration:
683.822489ms
FAIL (2025-10-31 21:25:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 683.842907ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [........X.................................X...........X.....] duration:
476.084491ms
FAIL (2025-10-31 21:25:19 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 476.096152ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X.....................X......] duration:
466.447319ms
FAIL (2025-10-31 21:25:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:22:00.944281574 (106h8m27.645226459s) [upload=93401]: Starting upload
Oct 31 21:22:00.944356304 (106h8m27.645301199s) [upload=93401]: Done starting upload
Oct 31 21:22:00.944468063 (106h8m27.645412948s) [upload=93401]: Uploading segments...
Oct 31 21:22:00.944630548 (106h8m27.645575433s) [upload=93401]: Got next segment. Inline: false
Oct 31 21:22:00.944665213 (106h8m27.645610108s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:00.944670022 (106h8m27.645614907s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:00.963897347 (106h8m27.664842243s) [seg_pos={0 0}, upload=93401]: Begin upload segment...
Oct 31 21:22:00.963998978 (106h8m27.664943873s) [seg_pos={0 0}, upload=93401]: Done begin upload segment.
Oct 31 21:22:00.964028473 (106h8m27.664973368s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964037450 (106h8m27.664982345s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964444233 (106h8m27.665389128s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964892844 (106h8m27.665837739s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965325144 (106h8m27.666270039s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965625197 (106h8m27.666570082s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966019857 (106h8m27.666964752s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966370855 (106h8m27.66731574s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966687539 (106h8m27.667632424s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.967022156 (106h8m27.667967042s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.968385533 (106h8m27.669330428s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.969631550 (106h8m27.670576435s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.971122787 (106h8m27.672067672s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.972500710 (106h8m27.673445595s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.973763789 (106h8m27.674708684s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.974994406 (106h8m27.675939291s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:01.045346730 (106h8m27.746291626s) [upload=93401]: Next returned nil segment
Oct 31 21:22:01.045360827 (106h8m27.746305712s) [upload=93401]: Waiting for error group managing segments...
Oct 31 21:22:01.055543281 (106h8m27.756488176s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055753104 (106h8m27.75669798s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055999196 (106h8m27.756944081s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056605764 (106h8m27.757550649s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056939789 (106h8m27.757884675s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057268406 (106h8m27.758213291s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057510931 (106h8m27.758455815s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057806124 (106h8m27.758751009s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.058699179 (106h8m27.759644064s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059036802 (106h8m27.759981688s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059484752 (106h8m27.760429636s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:30.929275627 (106h8m57.630220522s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929342072 (106h8m57.630286956s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:31798->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929405521 (106h8m57.630350416s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929460374 (106h8m57.630405269s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:53362->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929546896 (106h8m57.630491791s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 5.161.89.252:26862->77.90.31.19:4841: use of closed network connection; context deadline exceeded
Oct 31 21:22:30.929573285 (106h8m57.63051818s) [seg_pos={0 0}, upload=93401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:22:30.950873398 (106h8m57.651818293s) [upload=93401]: Done uploading segments...
Oct 31 21:22:30.950890520 (106h8m57.651835415s) [upload=93401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 466.470472ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........X...........................................X.....] duration:
670.015356ms
FAIL (2025-10-31 21:25:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 670.039872ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............X................X.............X..............] duration:
381.249856ms
FAIL (2025-10-31 21:25:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 381.269944ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.........X.................................X...........X....] duration:
510.844203ms
FAIL (2025-10-31 21:25:02 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 510.860314ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........X...........................................X....] duration:
629.581857ms
FAIL (2025-10-31 21:24:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 629.59399ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............................X............X..........X.X..] duration:
1.682770498s
FAIL (2025-10-31 21:24:31 UTC)
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
K3VNVMW4D5ZS4QLA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.936453020 (106h9m38.637397935s) [noise=true , address=45.91.116.4:4943 , piece_id=243RKZW4BEML4R2FFSYQJPAMWH4CNEPLOA6HNF2QE34KZRDPNXMQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.936945263 (106h9m38.637890188s) [noise=true , address=77.90.31.12:4599 , piece_id=U74XV7GO2KBUCAAS7TM2X5B7HPRU65Z7OFVBAHH3FSEM3RD4UEWQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.937417759 (106h9m38.638362674s) [noise=true , address=45.91.116.7:4034 , piece_id=W5M5KDA5ZUBO6NH3ZDA6DDGZJVCRNOSC5GQO6ES6T52DIXGFRHQQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.937844629 (106h9m38.638789544s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.938276769 (106h9m38.639221684s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.938759274 (106h9m38.6397042s) [noise=true , address=45.91.116.4:4963 , piece_id=B6QJWTQOHC7V7EEWZ2MLXH7DXMGGDK5AMX6WNNGO5TVKJJTEWMVA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.939226250 (106h9m38.640171165s) [noise=true , address=45.91.116.5:4326 , piece_id=3DCJH36YX5NCWBCI7WPI6XRRCIAZLF4L64BAMUHCOJL5LDXHST2Q, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.939663400 (106h9m38.640608315s) [noise=true , address=145.79.227.38:4998 , piece_id=QOYPHU6BDTLOMIA2MOBQVA6DEKTRKTJS4XEMTUWYFJ2GCWTBCFGQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.940657263 (106h9m38.641602178s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.941228184 (106h9m38.642173098s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.946730872 (106h9m38.647675787s) [noise=true , address=77.90.31.16:4759 , piece_id=QHJCJ5YSBE6CPMVIQBNOMGGVIGHJ2OWWVMOLPL5SQVJ4UFZI24PQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.948817905 (106h9m38.649762821s) [noise=true , address=45.91.116.13:4160 , piece_id=JGTRITASZGTS2FEJBR5M2LDDJ36PKYR3EFJMAH6CJ4JTMTZUZE2Q, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.950060556 (106h9m38.651005471s) [noise=true , address=45.91.116.10:4085 , piece_id=FGQTDJBIGMXW2F7FHJSJ7H5RYCKGCEIPE44E3DI6CJTQZWKOEVRA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.956241747 (106h9m38.657186672s) [noise=true , address=145.79.227.43:4163 , piece_id=OXKUQ6IF564LJIUTUKSBL6XEJ23GUTE5ALJCBLLRN42M5YNWIUCQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.958515621 (106h9m38.659460537s) [noise=true , address=145.79.227.38:4006 , piece_id=F2WDTLGHA4RWUTMZQIOABFHNFULMX2D355MEZJNFMUU57ATGWACQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.961843962 (106h9m38.662788887s) [noise=true , address=145.79.227.13:4836 , piece_id=K2XFNVC3RXKXEBF4GTNU56B5H6YTEBOCKVBKQV2PSVYMBME5A2XQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:12.063468287 (106h9m38.764413202s) [upload=93420]: Next returned nil segment
Oct 31 21:23:12.063488516 (106h9m38.76443343s) [upload=93420]: Waiting for error group managing segments...
Oct 31 21:23:12.075808427 (106h9m38.776753342s) [noise=true , address=145.79.227.43:4163 , piece_id=OXKUQ6IF564LJIUTUKSBL6XEJ23GUTE5ALJCBLLRN42M5YNWIUCQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.105232644 (106h9m38.806177549s) [noise=true , address=145.79.227.38:4998 , piece_id=QOYPHU6BDTLOMIA2MOBQVA6DEKTRKTJS4XEMTUWYFJ2GCWTBCFGQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.107710410 (106h9m38.808655325s) [noise=true , address=145.79.227.13:4836 , piece_id=K2XFNVC3RXKXEBF4GTNU56B5H6YTEBOCKVBKQV2PSVYMBME5A2XQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.118028529 (106h9m38.818973444s) [noise=true , address=45.91.116.4:4943 , piece_id=243RKZW4BEML4R2FFSYQJPAMWH4CNEPLOA6HNF2QE34KZRDPNXMQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.137402949 (106h9m38.838347955s) [noise=true , address=45.91.116.7:4034 , piece_id=W5M5KDA5ZUBO6NH3ZDA6DDGZJVCRNOSC5GQO6ES6T52DIXGFRHQQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.278017809 (106h9m38.978962695s) [noise=true , address=45.91.116.5:4326 , piece_id=3DCJH36YX5NCWBCI7WPI6XRRCIAZLF4L64BAMUHCOJL5LDXHST2Q, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.333784943 (106h9m39.034729828s) [noise=true , address=45.91.116.10:4085 , piece_id=FGQTDJBIGMXW2F7FHJSJ7H5RYCKGCEIPE44E3DI6CJTQZWKOEVRA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.367267139 (106h9m39.068212034s) [noise=true , address=45.91.116.8:4072 , piece_id=77IRSHYY2UV2GKIV2GLS63G4SP2GKTCAYCT6K3VNVMW4D5ZS4QLA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.400124803 (106h9m39.101069718s) [noise=true , address=45.91.116.4:4963 , piece_id=B6QJWTQOHC7V7EEWZ2MLXH7DXMGGDK5AMX6WNNGO5TVKJJTEWMVA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.573123492 (106h9m39.274068387s) [noise=true , address=145.79.227.38:4006 , piece_id=F2WDTLGHA4RWUTMZQIOABFHNFULMX2D355MEZJNFMUU57ATGWACQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.617098341 (106h9m39.318043227s) [noise=true , address=45.91.116.13:4160 , piece_id=JGTRITASZGTS2FEJBR5M2LDDJ36PKYR3EFJMAH6CJ4JTMTZUZE2Q, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:13.345716310 (106h9m40.046661225s) [noise=true , address=77.90.31.12:4599 , piece_id=U74XV7GO2KBUCAAS7TM2X5B7HPRU65Z7OFVBAHH3FSEM3RD4UEWQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:13.827713328 (106h9m40.528658223s) [noise=true , address=77.90.31.16:4759 , piece_id=QHJCJ5YSBE6CPMVIQBNOMGGVIGHJ2OWWVMOLPL5SQVJ4UFZI24PQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:14.056029484 (106h9m40.75697437s) [noise=true , address=77.90.31.21:4911 , piece_id=BQVGAE7XLX3KVML7PAB5XCHU5PUQBQHCTUAJJ2D5HIQYAGWFWSRQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:14.056046707 (106h9m40.756991591s) [seg_pos={0 0}, upload=93420]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:23:14.056146393 (106h9m40.757091279s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF
Oct 31 21:23:14.056153627 (106h9m40.757098512s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056437710 (106h9m40.757382595s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k29MGpez3ByMHnG8qGBm9iPwo3VrKUGDHpz9CqGEGNn47rxDx): protocol: rpc: dial tcp 77.90.31.14:4679: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:23:14.056445925 (106h9m40.75739081s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056536505 (106h9m40.757481441s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF
Oct 31 21:23:14.056543578 (106h9m40.757488483s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056652873 (106h9m40.757597768s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; write tcp 5.161.89.252:34960->77.90.31.19:4862: use of closed network connection; write tcp 5.161.89.252:34960->77.90.31.19:4862: use of closed network connection; EOF
Oct 31 21:23:14.056660638 (106h9m40.757605523s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056676818 (106h9m40.757621693s) [seg_pos={0 0}, upload=93420]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:23:14.056740047 (106h9m40.757684932s) [upload=93420]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:23:14.056746409 (106h9m40.757691294s) [upload=93420]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:23:14.097207661 (106h9m40.798152547s) [upload=93420]: Done uploading segments...
Oct 31 21:23:14.097221608 (106h9m40.798166482s) [upload=93420]: Upload finished. err: <nil>
Duration: 1.68278782s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............X................X.............X.............] duration:
626.283934ms
FAIL (2025-10-31 21:24:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 626.30336ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............X...........................................X...] duration:
25.725424843s
FAIL (2025-10-31 21:24:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 25.725443929s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................X.....................X.....] duration:
366.308269ms
FAIL (2025-10-31 21:24:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:22:00.944281574 (106h8m27.645226459s) [upload=93401]: Starting upload
Oct 31 21:22:00.944356304 (106h8m27.645301199s) [upload=93401]: Done starting upload
Oct 31 21:22:00.944468063 (106h8m27.645412948s) [upload=93401]: Uploading segments...
Oct 31 21:22:00.944630548 (106h8m27.645575433s) [upload=93401]: Got next segment. Inline: false
Oct 31 21:22:00.944665213 (106h8m27.645610108s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:00.944670022 (106h8m27.645614907s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:00.963897347 (106h8m27.664842243s) [seg_pos={0 0}, upload=93401]: Begin upload segment...
Oct 31 21:22:00.963998978 (106h8m27.664943873s) [seg_pos={0 0}, upload=93401]: Done begin upload segment.
Oct 31 21:22:00.964028473 (106h8m27.664973368s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964037450 (106h8m27.664982345s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964444233 (106h8m27.665389128s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964892844 (106h8m27.665837739s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965325144 (106h8m27.666270039s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965625197 (106h8m27.666570082s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966019857 (106h8m27.666964752s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966370855 (106h8m27.66731574s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966687539 (106h8m27.667632424s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.967022156 (106h8m27.667967042s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.968385533 (106h8m27.669330428s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.969631550 (106h8m27.670576435s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.971122787 (106h8m27.672067672s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.972500710 (106h8m27.673445595s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.973763789 (106h8m27.674708684s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.974994406 (106h8m27.675939291s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:01.045346730 (106h8m27.746291626s) [upload=93401]: Next returned nil segment
Oct 31 21:22:01.045360827 (106h8m27.746305712s) [upload=93401]: Waiting for error group managing segments...
Oct 31 21:22:01.055543281 (106h8m27.756488176s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055753104 (106h8m27.75669798s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055999196 (106h8m27.756944081s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056605764 (106h8m27.757550649s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056939789 (106h8m27.757884675s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057268406 (106h8m27.758213291s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057510931 (106h8m27.758455815s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057806124 (106h8m27.758751009s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.058699179 (106h8m27.759644064s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059036802 (106h8m27.759981688s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059484752 (106h8m27.760429636s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:30.929275627 (106h8m57.630220522s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929342072 (106h8m57.630286956s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:31798->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929405521 (106h8m57.630350416s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929460374 (106h8m57.630405269s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:53362->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929546896 (106h8m57.630491791s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 5.161.89.252:26862->77.90.31.19:4841: use of closed network connection; context deadline exceeded
Oct 31 21:22:30.929573285 (106h8m57.63051818s) [seg_pos={0 0}, upload=93401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:22:30.950873398 (106h8m57.651818293s) [upload=93401]: Done uploading segments...
Oct 31 21:22:30.950890520 (106h8m57.651835415s) [upload=93401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 366.342122ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [..........X.................................X...........X...] duration:
1.045926325s
FAIL (2025-10-31 21:24:04 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.045947114s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X................X.............X............] duration:
1.482035465s
FAIL (2025-10-31 21:24:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.482054731s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [................................X............X..........X.X.] duration:
2.472281766s
FAIL (2025-10-31 21:23:58 UTC)
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
K3VNVMW4D5ZS4QLA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.936453020 (106h9m38.637397935s) [noise=true , address=45.91.116.4:4943 , piece_id=243RKZW4BEML4R2FFSYQJPAMWH4CNEPLOA6HNF2QE34KZRDPNXMQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.936945263 (106h9m38.637890188s) [noise=true , address=77.90.31.12:4599 , piece_id=U74XV7GO2KBUCAAS7TM2X5B7HPRU65Z7OFVBAHH3FSEM3RD4UEWQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.937417759 (106h9m38.638362674s) [noise=true , address=45.91.116.7:4034 , piece_id=W5M5KDA5ZUBO6NH3ZDA6DDGZJVCRNOSC5GQO6ES6T52DIXGFRHQQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.937844629 (106h9m38.638789544s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.938276769 (106h9m38.639221684s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.938759274 (106h9m38.6397042s) [noise=true , address=45.91.116.4:4963 , piece_id=B6QJWTQOHC7V7EEWZ2MLXH7DXMGGDK5AMX6WNNGO5TVKJJTEWMVA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.939226250 (106h9m38.640171165s) [noise=true , address=45.91.116.5:4326 , piece_id=3DCJH36YX5NCWBCI7WPI6XRRCIAZLF4L64BAMUHCOJL5LDXHST2Q, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.939663400 (106h9m38.640608315s) [noise=true , address=145.79.227.38:4998 , piece_id=QOYPHU6BDTLOMIA2MOBQVA6DEKTRKTJS4XEMTUWYFJ2GCWTBCFGQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.940657263 (106h9m38.641602178s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.941228184 (106h9m38.642173098s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.946730872 (106h9m38.647675787s) [noise=true , address=77.90.31.16:4759 , piece_id=QHJCJ5YSBE6CPMVIQBNOMGGVIGHJ2OWWVMOLPL5SQVJ4UFZI24PQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.948817905 (106h9m38.649762821s) [noise=true , address=45.91.116.13:4160 , piece_id=JGTRITASZGTS2FEJBR5M2LDDJ36PKYR3EFJMAH6CJ4JTMTZUZE2Q, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.950060556 (106h9m38.651005471s) [noise=true , address=45.91.116.10:4085 , piece_id=FGQTDJBIGMXW2F7FHJSJ7H5RYCKGCEIPE44E3DI6CJTQZWKOEVRA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.956241747 (106h9m38.657186672s) [noise=true , address=145.79.227.43:4163 , piece_id=OXKUQ6IF564LJIUTUKSBL6XEJ23GUTE5ALJCBLLRN42M5YNWIUCQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.958515621 (106h9m38.659460537s) [noise=true , address=145.79.227.38:4006 , piece_id=F2WDTLGHA4RWUTMZQIOABFHNFULMX2D355MEZJNFMUU57ATGWACQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.961843962 (106h9m38.662788887s) [noise=true , address=145.79.227.13:4836 , piece_id=K2XFNVC3RXKXEBF4GTNU56B5H6YTEBOCKVBKQV2PSVYMBME5A2XQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:12.063468287 (106h9m38.764413202s) [upload=93420]: Next returned nil segment
Oct 31 21:23:12.063488516 (106h9m38.76443343s) [upload=93420]: Waiting for error group managing segments...
Oct 31 21:23:12.075808427 (106h9m38.776753342s) [noise=true , address=145.79.227.43:4163 , piece_id=OXKUQ6IF564LJIUTUKSBL6XEJ23GUTE5ALJCBLLRN42M5YNWIUCQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.105232644 (106h9m38.806177549s) [noise=true , address=145.79.227.38:4998 , piece_id=QOYPHU6BDTLOMIA2MOBQVA6DEKTRKTJS4XEMTUWYFJ2GCWTBCFGQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.107710410 (106h9m38.808655325s) [noise=true , address=145.79.227.13:4836 , piece_id=K2XFNVC3RXKXEBF4GTNU56B5H6YTEBOCKVBKQV2PSVYMBME5A2XQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.118028529 (106h9m38.818973444s) [noise=true , address=45.91.116.4:4943 , piece_id=243RKZW4BEML4R2FFSYQJPAMWH4CNEPLOA6HNF2QE34KZRDPNXMQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.137402949 (106h9m38.838347955s) [noise=true , address=45.91.116.7:4034 , piece_id=W5M5KDA5ZUBO6NH3ZDA6DDGZJVCRNOSC5GQO6ES6T52DIXGFRHQQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.278017809 (106h9m38.978962695s) [noise=true , address=45.91.116.5:4326 , piece_id=3DCJH36YX5NCWBCI7WPI6XRRCIAZLF4L64BAMUHCOJL5LDXHST2Q, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.333784943 (106h9m39.034729828s) [noise=true , address=45.91.116.10:4085 , piece_id=FGQTDJBIGMXW2F7FHJSJ7H5RYCKGCEIPE44E3DI6CJTQZWKOEVRA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.367267139 (106h9m39.068212034s) [noise=true , address=45.91.116.8:4072 , piece_id=77IRSHYY2UV2GKIV2GLS63G4SP2GKTCAYCT6K3VNVMW4D5ZS4QLA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.400124803 (106h9m39.101069718s) [noise=true , address=45.91.116.4:4963 , piece_id=B6QJWTQOHC7V7EEWZ2MLXH7DXMGGDK5AMX6WNNGO5TVKJJTEWMVA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.573123492 (106h9m39.274068387s) [noise=true , address=145.79.227.38:4006 , piece_id=F2WDTLGHA4RWUTMZQIOABFHNFULMX2D355MEZJNFMUU57ATGWACQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.617098341 (106h9m39.318043227s) [noise=true , address=45.91.116.13:4160 , piece_id=JGTRITASZGTS2FEJBR5M2LDDJ36PKYR3EFJMAH6CJ4JTMTZUZE2Q, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:13.345716310 (106h9m40.046661225s) [noise=true , address=77.90.31.12:4599 , piece_id=U74XV7GO2KBUCAAS7TM2X5B7HPRU65Z7OFVBAHH3FSEM3RD4UEWQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:13.827713328 (106h9m40.528658223s) [noise=true , address=77.90.31.16:4759 , piece_id=QHJCJ5YSBE6CPMVIQBNOMGGVIGHJ2OWWVMOLPL5SQVJ4UFZI24PQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:14.056029484 (106h9m40.75697437s) [noise=true , address=77.90.31.21:4911 , piece_id=BQVGAE7XLX3KVML7PAB5XCHU5PUQBQHCTUAJJ2D5HIQYAGWFWSRQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:14.056046707 (106h9m40.756991591s) [seg_pos={0 0}, upload=93420]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:23:14.056146393 (106h9m40.757091279s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF
Oct 31 21:23:14.056153627 (106h9m40.757098512s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056437710 (106h9m40.757382595s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k29MGpez3ByMHnG8qGBm9iPwo3VrKUGDHpz9CqGEGNn47rxDx): protocol: rpc: dial tcp 77.90.31.14:4679: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:23:14.056445925 (106h9m40.75739081s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056536505 (106h9m40.757481441s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF
Oct 31 21:23:14.056543578 (106h9m40.757488483s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056652873 (106h9m40.757597768s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; write tcp 5.161.89.252:34960->77.90.31.19:4862: use of closed network connection; write tcp 5.161.89.252:34960->77.90.31.19:4862: use of closed network connection; EOF
Oct 31 21:23:14.056660638 (106h9m40.757605523s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056676818 (106h9m40.757621693s) [seg_pos={0 0}, upload=93420]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:23:14.056740047 (106h9m40.757684932s) [upload=93420]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:23:14.056746409 (106h9m40.757691294s) [upload=93420]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:23:14.097207661 (106h9m40.798152547s) [upload=93420]: Done uploading segments...
Oct 31 21:23:14.097221608 (106h9m40.798166482s) [upload=93420]: Upload finished. err: <nil>
Duration: 2.472307283s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X.....................X....] duration:
428.54223ms
FAIL (2025-10-31 21:23:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:22:00.944281574 (106h8m27.645226459s) [upload=93401]: Starting upload
Oct 31 21:22:00.944356304 (106h8m27.645301199s) [upload=93401]: Done starting upload
Oct 31 21:22:00.944468063 (106h8m27.645412948s) [upload=93401]: Uploading segments...
Oct 31 21:22:00.944630548 (106h8m27.645575433s) [upload=93401]: Got next segment. Inline: false
Oct 31 21:22:00.944665213 (106h8m27.645610108s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:00.944670022 (106h8m27.645614907s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:00.963897347 (106h8m27.664842243s) [seg_pos={0 0}, upload=93401]: Begin upload segment...
Oct 31 21:22:00.963998978 (106h8m27.664943873s) [seg_pos={0 0}, upload=93401]: Done begin upload segment.
Oct 31 21:22:00.964028473 (106h8m27.664973368s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964037450 (106h8m27.664982345s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964444233 (106h8m27.665389128s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964892844 (106h8m27.665837739s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965325144 (106h8m27.666270039s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965625197 (106h8m27.666570082s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966019857 (106h8m27.666964752s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966370855 (106h8m27.66731574s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966687539 (106h8m27.667632424s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.967022156 (106h8m27.667967042s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.968385533 (106h8m27.669330428s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.969631550 (106h8m27.670576435s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.971122787 (106h8m27.672067672s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.972500710 (106h8m27.673445595s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.973763789 (106h8m27.674708684s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.974994406 (106h8m27.675939291s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:01.045346730 (106h8m27.746291626s) [upload=93401]: Next returned nil segment
Oct 31 21:22:01.045360827 (106h8m27.746305712s) [upload=93401]: Waiting for error group managing segments...
Oct 31 21:22:01.055543281 (106h8m27.756488176s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055753104 (106h8m27.75669798s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055999196 (106h8m27.756944081s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056605764 (106h8m27.757550649s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056939789 (106h8m27.757884675s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057268406 (106h8m27.758213291s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057510931 (106h8m27.758455815s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057806124 (106h8m27.758751009s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.058699179 (106h8m27.759644064s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059036802 (106h8m27.759981688s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059484752 (106h8m27.760429636s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:30.929275627 (106h8m57.630220522s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929342072 (106h8m57.630286956s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:31798->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929405521 (106h8m57.630350416s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929460374 (106h8m57.630405269s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:53362->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929546896 (106h8m57.630491791s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 5.161.89.252:26862->77.90.31.19:4841: use of closed network connection; context deadline exceeded
Oct 31 21:22:30.929573285 (106h8m57.63051818s) [seg_pos={0 0}, upload=93401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:22:30.950873398 (106h8m57.651818293s) [upload=93401]: Done uploading segments...
Oct 31 21:22:30.950890520 (106h8m57.651835415s) [upload=93401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 428.560545ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................................X............X..........X.X] duration:
27.392097041s
FAIL (2025-10-31 21:23:39 UTC)
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
K3VNVMW4D5ZS4QLA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.936453020 (106h9m38.637397935s) [noise=true , address=45.91.116.4:4943 , piece_id=243RKZW4BEML4R2FFSYQJPAMWH4CNEPLOA6HNF2QE34KZRDPNXMQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.936945263 (106h9m38.637890188s) [noise=true , address=77.90.31.12:4599 , piece_id=U74XV7GO2KBUCAAS7TM2X5B7HPRU65Z7OFVBAHH3FSEM3RD4UEWQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.937417759 (106h9m38.638362674s) [noise=true , address=45.91.116.7:4034 , piece_id=W5M5KDA5ZUBO6NH3ZDA6DDGZJVCRNOSC5GQO6ES6T52DIXGFRHQQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.937844629 (106h9m38.638789544s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.938276769 (106h9m38.639221684s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.938759274 (106h9m38.6397042s) [noise=true , address=45.91.116.4:4963 , piece_id=B6QJWTQOHC7V7EEWZ2MLXH7DXMGGDK5AMX6WNNGO5TVKJJTEWMVA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.939226250 (106h9m38.640171165s) [noise=true , address=45.91.116.5:4326 , piece_id=3DCJH36YX5NCWBCI7WPI6XRRCIAZLF4L64BAMUHCOJL5LDXHST2Q, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.939663400 (106h9m38.640608315s) [noise=true , address=145.79.227.38:4998 , piece_id=QOYPHU6BDTLOMIA2MOBQVA6DEKTRKTJS4XEMTUWYFJ2GCWTBCFGQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.940657263 (106h9m38.641602178s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.941228184 (106h9m38.642173098s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.946730872 (106h9m38.647675787s) [noise=true , address=77.90.31.16:4759 , piece_id=QHJCJ5YSBE6CPMVIQBNOMGGVIGHJ2OWWVMOLPL5SQVJ4UFZI24PQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.948817905 (106h9m38.649762821s) [noise=true , address=45.91.116.13:4160 , piece_id=JGTRITASZGTS2FEJBR5M2LDDJ36PKYR3EFJMAH6CJ4JTMTZUZE2Q, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.950060556 (106h9m38.651005471s) [noise=true , address=45.91.116.10:4085 , piece_id=FGQTDJBIGMXW2F7FHJSJ7H5RYCKGCEIPE44E3DI6CJTQZWKOEVRA, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.956241747 (106h9m38.657186672s) [noise=true , address=145.79.227.43:4163 , piece_id=OXKUQ6IF564LJIUTUKSBL6XEJ23GUTE5ALJCBLLRN42M5YNWIUCQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.958515621 (106h9m38.659460537s) [noise=true , address=145.79.227.38:4006 , piece_id=F2WDTLGHA4RWUTMZQIOABFHNFULMX2D355MEZJNFMUU57ATGWACQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:11.961843962 (106h9m38.662788887s) [noise=true , address=145.79.227.13:4836 , piece_id=K2XFNVC3RXKXEBF4GTNU56B5H6YTEBOCKVBKQV2PSVYMBME5A2XQ, seg_pos={0 0}, upload=93420]: Uploading piece...
Oct 31 21:23:12.063468287 (106h9m38.764413202s) [upload=93420]: Next returned nil segment
Oct 31 21:23:12.063488516 (106h9m38.76443343s) [upload=93420]: Waiting for error group managing segments...
Oct 31 21:23:12.075808427 (106h9m38.776753342s) [noise=true , address=145.79.227.43:4163 , piece_id=OXKUQ6IF564LJIUTUKSBL6XEJ23GUTE5ALJCBLLRN42M5YNWIUCQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.105232644 (106h9m38.806177549s) [noise=true , address=145.79.227.38:4998 , piece_id=QOYPHU6BDTLOMIA2MOBQVA6DEKTRKTJS4XEMTUWYFJ2GCWTBCFGQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.107710410 (106h9m38.808655325s) [noise=true , address=145.79.227.13:4836 , piece_id=K2XFNVC3RXKXEBF4GTNU56B5H6YTEBOCKVBKQV2PSVYMBME5A2XQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.118028529 (106h9m38.818973444s) [noise=true , address=45.91.116.4:4943 , piece_id=243RKZW4BEML4R2FFSYQJPAMWH4CNEPLOA6HNF2QE34KZRDPNXMQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.137402949 (106h9m38.838347955s) [noise=true , address=45.91.116.7:4034 , piece_id=W5M5KDA5ZUBO6NH3ZDA6DDGZJVCRNOSC5GQO6ES6T52DIXGFRHQQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.278017809 (106h9m38.978962695s) [noise=true , address=45.91.116.5:4326 , piece_id=3DCJH36YX5NCWBCI7WPI6XRRCIAZLF4L64BAMUHCOJL5LDXHST2Q, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.333784943 (106h9m39.034729828s) [noise=true , address=45.91.116.10:4085 , piece_id=FGQTDJBIGMXW2F7FHJSJ7H5RYCKGCEIPE44E3DI6CJTQZWKOEVRA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.367267139 (106h9m39.068212034s) [noise=true , address=45.91.116.8:4072 , piece_id=77IRSHYY2UV2GKIV2GLS63G4SP2GKTCAYCT6K3VNVMW4D5ZS4QLA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.400124803 (106h9m39.101069718s) [noise=true , address=45.91.116.4:4963 , piece_id=B6QJWTQOHC7V7EEWZ2MLXH7DXMGGDK5AMX6WNNGO5TVKJJTEWMVA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.573123492 (106h9m39.274068387s) [noise=true , address=145.79.227.38:4006 , piece_id=F2WDTLGHA4RWUTMZQIOABFHNFULMX2D355MEZJNFMUU57ATGWACQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:12.617098341 (106h9m39.318043227s) [noise=true , address=45.91.116.13:4160 , piece_id=JGTRITASZGTS2FEJBR5M2LDDJ36PKYR3EFJMAH6CJ4JTMTZUZE2Q, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:13.345716310 (106h9m40.046661225s) [noise=true , address=77.90.31.12:4599 , piece_id=U74XV7GO2KBUCAAS7TM2X5B7HPRU65Z7OFVBAHH3FSEM3RD4UEWQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:13.827713328 (106h9m40.528658223s) [noise=true , address=77.90.31.16:4759 , piece_id=QHJCJ5YSBE6CPMVIQBNOMGGVIGHJ2OWWVMOLPL5SQVJ4UFZI24PQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:14.056029484 (106h9m40.75697437s) [noise=true , address=77.90.31.21:4911 , piece_id=BQVGAE7XLX3KVML7PAB5XCHU5PUQBQHCTUAJJ2D5HIQYAGWFWSRQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: <nil>
Oct 31 21:23:14.056046707 (106h9m40.756991591s) [seg_pos={0 0}, upload=93420]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:23:14.056146393 (106h9m40.757091279s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF
Oct 31 21:23:14.056153627 (106h9m40.757098512s) [noise=true , address=77.90.31.18:4834 , piece_id=NYHSDT45KWMRXF2IGSMGHMDMIUXDYW3RQOVT3FI535GUUJIMPMLQ, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056437710 (106h9m40.757382595s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k29MGpez3ByMHnG8qGBm9iPwo3VrKUGDHpz9CqGEGNn47rxDx): protocol: rpc: dial tcp 77.90.31.14:4679: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:23:14.056445925 (106h9m40.75739081s) [noise=true , address=77.90.31.14:4679 , piece_id=DXPGOIKY2JW7K3PL6HQOSRFIGZXAORHOR644XI455OHU7CJVXWYA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056536505 (106h9m40.757481441s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF
Oct 31 21:23:14.056543578 (106h9m40.757488483s) [noise=true , address=77.90.31.15:4709 , piece_id=D74BPJO7IROYPDSEOF3E72ABREWATJRUPLCH5QFKGVEJVHZPHLXA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056652873 (106h9m40.757597768s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; write tcp 5.161.89.252:34960->77.90.31.19:4862: use of closed network connection; write tcp 5.161.89.252:34960->77.90.31.19:4862: use of closed network connection; EOF
Oct 31 21:23:14.056660638 (106h9m40.757605523s) [noise=true , address=77.90.31.19:4862 , piece_id=YRK2IR66S3BR2GRQBXU4X5JUTWD6UFTKJWWTZGNZ5HF7JQG267WA, seg_pos={0 0}, upload=93420]: Long tail cancellation due to optimal threshold
Oct 31 21:23:14.056676818 (106h9m40.757621693s) [seg_pos={0 0}, upload=93420]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:23:14.056740047 (106h9m40.757684932s) [upload=93420]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:23:14.056746409 (106h9m40.757691294s) [upload=93420]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:23:14.097207661 (106h9m40.798152547s) [upload=93420]: Done uploading segments...
Oct 31 21:23:14.097221608 (106h9m40.798166482s) [upload=93420]: Upload finished. err: <nil>
Duration: 27.39214502s
core
uplink
us-select-1
us1-placement-20
|