Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
513.383351ms
FAIL (2025-10-31 21:08:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 513.418416ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
417.804404ms
FAIL (2025-10-31 21:08:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
tBegin
Oct 31 21:03:12.217456618 (105h49m38.918401503s) [upload=93134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:12.238821717 (105h49m38.939766612s) [seg_pos={0 0}, upload=93134]: Begin upload segment...
Oct 31 21:03:12.238864187 (105h49m38.939809082s) [seg_pos={0 0}, upload=93134]: Done begin upload segment.
Oct 31 21:03:12.238885798 (105h49m38.939830683s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.238936744 (105h49m38.939881628s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239432864 (105h49m38.940377749s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239474923 (105h49m38.940419818s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239689976 (105h49m38.940634861s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239937421 (105h49m38.940882305s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239946858 (105h49m38.940891753s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240172622 (105h49m38.941117517s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240219950 (105h49m38.941164835s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240415367 (105h49m38.941360262s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240640008 (105h49m38.941584893s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240714867 (105h49m38.941659752s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240874156 (105h49m38.941819041s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241069643 (105h49m38.942014538s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241168208 (105h49m38.942113103s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241718780 (105h49m38.942663665s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241812185 (105h49m38.942757081s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.244034422 (105h49m38.944979318s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.342147728 (105h49m39.043092643s) [upload=93134]: Next returned nil segment
Oct 31 21:03:12.342173847 (105h49m39.043118802s) [upload=93134]: Waiting for error group managing segments...
Oct 31 21:03:12.352785595 (105h49m39.053730511s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.352905409 (105h49m39.053850304s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.365098953 (105h49m39.066043858s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.367756977 (105h49m39.068701892s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.372646285 (105h49m39.07359121s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.374120059 (105h49m39.075064973s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.378605118 (105h49m39.079550023s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.550417419 (105h49m39.251362304s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.875337427 (105h49m39.576282322s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:37.185165129 (105h50m3.886110024s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:00.750493652 (105h50m27.451438537s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.070380024 (105h50m33.771324919s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.970385749 (105h50m34.671330644s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:12.205981337 (105h50m38.906926221s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206045126 (105h50m38.906990011s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12VfmooWvtic6UbcWtzBQ4ttYgUPbo8sjCJyM5VVCXgcQw6Ld6F, address:45.91.116.14:4205): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206074311 (105h50m38.907019196s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection
Oct 31 21:04:12.206096242 (105h50m38.907041117s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1YzWUkjkJXsP2Y5JYoFBM8Z8gsFbC5hzNGbWRgRTwg4uGge71p, address:45.91.116.6:4997): protocol: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection
Oct 31 21:04:12.206120908 (105h50m38.907065793s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: expected piece hash; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; EOF
Oct 31 21:04:12.206143972 (105h50m38.907088857s) [seg_pos={0 0}, upload=93134]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:04:12.226439497 (105h50m38.927384382s) [upload=93134]: Done uploading segments...
Oct 31 21:04:12.226460978 (105h50m38.927405873s) [upload=93134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 417.826576ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
594.881276ms
FAIL (2025-10-31 21:08:48 UTC)
orbiter warning: context deadline exceeded
Duration: 594.970292ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
441.375201ms
FAIL (2025-10-31 21:08:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
tBegin
Oct 31 21:03:12.217456618 (105h49m38.918401503s) [upload=93134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:12.238821717 (105h49m38.939766612s) [seg_pos={0 0}, upload=93134]: Begin upload segment...
Oct 31 21:03:12.238864187 (105h49m38.939809082s) [seg_pos={0 0}, upload=93134]: Done begin upload segment.
Oct 31 21:03:12.238885798 (105h49m38.939830683s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.238936744 (105h49m38.939881628s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239432864 (105h49m38.940377749s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239474923 (105h49m38.940419818s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239689976 (105h49m38.940634861s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239937421 (105h49m38.940882305s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239946858 (105h49m38.940891753s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240172622 (105h49m38.941117517s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240219950 (105h49m38.941164835s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240415367 (105h49m38.941360262s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240640008 (105h49m38.941584893s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240714867 (105h49m38.941659752s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240874156 (105h49m38.941819041s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241069643 (105h49m38.942014538s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241168208 (105h49m38.942113103s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241718780 (105h49m38.942663665s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241812185 (105h49m38.942757081s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.244034422 (105h49m38.944979318s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.342147728 (105h49m39.043092643s) [upload=93134]: Next returned nil segment
Oct 31 21:03:12.342173847 (105h49m39.043118802s) [upload=93134]: Waiting for error group managing segments...
Oct 31 21:03:12.352785595 (105h49m39.053730511s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.352905409 (105h49m39.053850304s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.365098953 (105h49m39.066043858s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.367756977 (105h49m39.068701892s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.372646285 (105h49m39.07359121s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.374120059 (105h49m39.075064973s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.378605118 (105h49m39.079550023s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.550417419 (105h49m39.251362304s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.875337427 (105h49m39.576282322s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:37.185165129 (105h50m3.886110024s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:00.750493652 (105h50m27.451438537s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.070380024 (105h50m33.771324919s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.970385749 (105h50m34.671330644s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:12.205981337 (105h50m38.906926221s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206045126 (105h50m38.906990011s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12VfmooWvtic6UbcWtzBQ4ttYgUPbo8sjCJyM5VVCXgcQw6Ld6F, address:45.91.116.14:4205): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206074311 (105h50m38.907019196s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection
Oct 31 21:04:12.206096242 (105h50m38.907041117s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1YzWUkjkJXsP2Y5JYoFBM8Z8gsFbC5hzNGbWRgRTwg4uGge71p, address:45.91.116.6:4997): protocol: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection
Oct 31 21:04:12.206120908 (105h50m38.907065793s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: expected piece hash; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; EOF
Oct 31 21:04:12.206143972 (105h50m38.907088857s) [seg_pos={0 0}, upload=93134]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:04:12.226439497 (105h50m38.927384382s) [upload=93134]: Done uploading segments...
Oct 31 21:04:12.226460978 (105h50m38.927405873s) [upload=93134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 441.397293ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X................X.......] duration:
363.760628ms
FAIL (2025-10-31 21:08:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 363.784082ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
382.961951ms
FAIL (2025-10-31 21:08:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 382.983963ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
583.019954ms
FAIL (2025-10-31 21:08:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 583.03881ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X................X......] duration:
340.817296ms
FAIL (2025-10-31 21:08:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 340.837704ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
838.835337ms
FAIL (2025-10-31 21:07:47 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 838.855515ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
624.866068ms
FAIL (2025-10-31 21:07:43 UTC)
orbiter warning: context deadline exceeded
Duration: 624.882639ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
541.655693ms
FAIL (2025-10-31 21:07:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 541.675989ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
423.019141ms
FAIL (2025-10-31 21:07:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
tBegin
Oct 31 21:03:12.217456618 (105h49m38.918401503s) [upload=93134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:12.238821717 (105h49m38.939766612s) [seg_pos={0 0}, upload=93134]: Begin upload segment...
Oct 31 21:03:12.238864187 (105h49m38.939809082s) [seg_pos={0 0}, upload=93134]: Done begin upload segment.
Oct 31 21:03:12.238885798 (105h49m38.939830683s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.238936744 (105h49m38.939881628s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239432864 (105h49m38.940377749s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239474923 (105h49m38.940419818s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239689976 (105h49m38.940634861s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239937421 (105h49m38.940882305s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239946858 (105h49m38.940891753s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240172622 (105h49m38.941117517s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240219950 (105h49m38.941164835s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240415367 (105h49m38.941360262s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240640008 (105h49m38.941584893s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240714867 (105h49m38.941659752s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240874156 (105h49m38.941819041s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241069643 (105h49m38.942014538s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241168208 (105h49m38.942113103s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241718780 (105h49m38.942663665s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241812185 (105h49m38.942757081s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.244034422 (105h49m38.944979318s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.342147728 (105h49m39.043092643s) [upload=93134]: Next returned nil segment
Oct 31 21:03:12.342173847 (105h49m39.043118802s) [upload=93134]: Waiting for error group managing segments...
Oct 31 21:03:12.352785595 (105h49m39.053730511s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.352905409 (105h49m39.053850304s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.365098953 (105h49m39.066043858s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.367756977 (105h49m39.068701892s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.372646285 (105h49m39.07359121s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.374120059 (105h49m39.075064973s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.378605118 (105h49m39.079550023s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.550417419 (105h49m39.251362304s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.875337427 (105h49m39.576282322s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:37.185165129 (105h50m3.886110024s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:00.750493652 (105h50m27.451438537s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.070380024 (105h50m33.771324919s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.970385749 (105h50m34.671330644s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:12.205981337 (105h50m38.906926221s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206045126 (105h50m38.906990011s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12VfmooWvtic6UbcWtzBQ4ttYgUPbo8sjCJyM5VVCXgcQw6Ld6F, address:45.91.116.14:4205): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206074311 (105h50m38.907019196s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection
Oct 31 21:04:12.206096242 (105h50m38.907041117s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1YzWUkjkJXsP2Y5JYoFBM8Z8gsFbC5hzNGbWRgRTwg4uGge71p, address:45.91.116.6:4997): protocol: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection
Oct 31 21:04:12.206120908 (105h50m38.907065793s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: expected piece hash; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; EOF
Oct 31 21:04:12.206143972 (105h50m38.907088857s) [seg_pos={0 0}, upload=93134]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:04:12.226439497 (105h50m38.927384382s) [upload=93134]: Done uploading segments...
Oct 31 21:04:12.226460978 (105h50m38.927405873s) [upload=93134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 423.041143ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
396.686566ms
FAIL (2025-10-31 21:07:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 396.710311ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X................X.....] duration:
350.032298ms
FAIL (2025-10-31 21:07:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 350.110172ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
449.931615ms
FAIL (2025-10-31 21:06:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 449.953306ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
862.155021ms
FAIL (2025-10-31 21:06:49 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 862.17563ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
546.637576ms
FAIL (2025-10-31 21:06:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 546.659166ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
447.233696ms
FAIL (2025-10-31 21:06:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 447.254454ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
665.581884ms
FAIL (2025-10-31 21:06:28 UTC)
orbiter warning: context deadline exceeded
Duration: 665.603825ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................X................X....] duration:
343.556103ms
FAIL (2025-10-31 21:06:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 343.70944ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
402.466664ms
FAIL (2025-10-31 21:06:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
tBegin
Oct 31 21:03:12.217456618 (105h49m38.918401503s) [upload=93134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:12.238821717 (105h49m38.939766612s) [seg_pos={0 0}, upload=93134]: Begin upload segment...
Oct 31 21:03:12.238864187 (105h49m38.939809082s) [seg_pos={0 0}, upload=93134]: Done begin upload segment.
Oct 31 21:03:12.238885798 (105h49m38.939830683s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.238936744 (105h49m38.939881628s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239432864 (105h49m38.940377749s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239474923 (105h49m38.940419818s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239689976 (105h49m38.940634861s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239937421 (105h49m38.940882305s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239946858 (105h49m38.940891753s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240172622 (105h49m38.941117517s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240219950 (105h49m38.941164835s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240415367 (105h49m38.941360262s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240640008 (105h49m38.941584893s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240714867 (105h49m38.941659752s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240874156 (105h49m38.941819041s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241069643 (105h49m38.942014538s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241168208 (105h49m38.942113103s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241718780 (105h49m38.942663665s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241812185 (105h49m38.942757081s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.244034422 (105h49m38.944979318s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.342147728 (105h49m39.043092643s) [upload=93134]: Next returned nil segment
Oct 31 21:03:12.342173847 (105h49m39.043118802s) [upload=93134]: Waiting for error group managing segments...
Oct 31 21:03:12.352785595 (105h49m39.053730511s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.352905409 (105h49m39.053850304s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.365098953 (105h49m39.066043858s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.367756977 (105h49m39.068701892s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.372646285 (105h49m39.07359121s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.374120059 (105h49m39.075064973s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.378605118 (105h49m39.079550023s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.550417419 (105h49m39.251362304s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.875337427 (105h49m39.576282322s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:37.185165129 (105h50m3.886110024s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:00.750493652 (105h50m27.451438537s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.070380024 (105h50m33.771324919s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.970385749 (105h50m34.671330644s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:12.205981337 (105h50m38.906926221s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206045126 (105h50m38.906990011s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12VfmooWvtic6UbcWtzBQ4ttYgUPbo8sjCJyM5VVCXgcQw6Ld6F, address:45.91.116.14:4205): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206074311 (105h50m38.907019196s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection
Oct 31 21:04:12.206096242 (105h50m38.907041117s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1YzWUkjkJXsP2Y5JYoFBM8Z8gsFbC5hzNGbWRgRTwg4uGge71p, address:45.91.116.6:4997): protocol: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection
Oct 31 21:04:12.206120908 (105h50m38.907065793s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: expected piece hash; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; EOF
Oct 31 21:04:12.206143972 (105h50m38.907088857s) [seg_pos={0 0}, upload=93134]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:04:12.226439497 (105h50m38.927384382s) [upload=93134]: Done uploading segments...
Oct 31 21:04:12.226460978 (105h50m38.927405873s) [upload=93134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 402.48054ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
508.072832ms
FAIL (2025-10-31 21:06:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 508.08838ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
442.492675ms
FAIL (2025-10-31 21:06:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
tBegin
Oct 31 21:03:12.217456618 (105h49m38.918401503s) [upload=93134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:12.238821717 (105h49m38.939766612s) [seg_pos={0 0}, upload=93134]: Begin upload segment...
Oct 31 21:03:12.238864187 (105h49m38.939809082s) [seg_pos={0 0}, upload=93134]: Done begin upload segment.
Oct 31 21:03:12.238885798 (105h49m38.939830683s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.238936744 (105h49m38.939881628s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239432864 (105h49m38.940377749s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239474923 (105h49m38.940419818s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239689976 (105h49m38.940634861s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239937421 (105h49m38.940882305s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239946858 (105h49m38.940891753s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240172622 (105h49m38.941117517s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240219950 (105h49m38.941164835s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240415367 (105h49m38.941360262s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240640008 (105h49m38.941584893s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240714867 (105h49m38.941659752s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240874156 (105h49m38.941819041s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241069643 (105h49m38.942014538s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241168208 (105h49m38.942113103s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241718780 (105h49m38.942663665s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241812185 (105h49m38.942757081s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.244034422 (105h49m38.944979318s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.342147728 (105h49m39.043092643s) [upload=93134]: Next returned nil segment
Oct 31 21:03:12.342173847 (105h49m39.043118802s) [upload=93134]: Waiting for error group managing segments...
Oct 31 21:03:12.352785595 (105h49m39.053730511s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.352905409 (105h49m39.053850304s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.365098953 (105h49m39.066043858s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.367756977 (105h49m39.068701892s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.372646285 (105h49m39.07359121s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.374120059 (105h49m39.075064973s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.378605118 (105h49m39.079550023s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.550417419 (105h49m39.251362304s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.875337427 (105h49m39.576282322s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:37.185165129 (105h50m3.886110024s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:00.750493652 (105h50m27.451438537s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.070380024 (105h50m33.771324919s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.970385749 (105h50m34.671330644s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:12.205981337 (105h50m38.906926221s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206045126 (105h50m38.906990011s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12VfmooWvtic6UbcWtzBQ4ttYgUPbo8sjCJyM5VVCXgcQw6Ld6F, address:45.91.116.14:4205): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206074311 (105h50m38.907019196s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection
Oct 31 21:04:12.206096242 (105h50m38.907041117s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1YzWUkjkJXsP2Y5JYoFBM8Z8gsFbC5hzNGbWRgRTwg4uGge71p, address:45.91.116.6:4997): protocol: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection
Oct 31 21:04:12.206120908 (105h50m38.907065793s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: expected piece hash; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; EOF
Oct 31 21:04:12.206143972 (105h50m38.907088857s) [seg_pos={0 0}, upload=93134]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:04:12.226439497 (105h50m38.927384382s) [upload=93134]: Done uploading segments...
Oct 31 21:04:12.226460978 (105h50m38.927405873s) [upload=93134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 442.512933ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
867.392564ms
FAIL (2025-10-31 21:06:04 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 867.408804ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
490.331907ms
FAIL (2025-10-31 21:06:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 490.352427ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
611.155511ms
FAIL (2025-10-31 21:06:01 UTC)
orbiter warning: context deadline exceeded
Duration: 611.36805ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X................X...] duration:
359.146987ms
FAIL (2025-10-31 21:05:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 359.165873ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
612.572177ms
FAIL (2025-10-31 21:05:38 UTC)
orbiter warning: context deadline exceeded
Duration: 612.591885ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
428.447589ms
FAIL (2025-10-31 21:05:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 428.488827ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
884.544676ms
FAIL (2025-10-31 21:05:31 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 884.565435ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
642.483262ms
FAIL (2025-10-31 21:05:20 UTC)
orbiter warning: context deadline exceeded
Duration: 642.511535ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X................X..] duration:
370.466231ms
FAIL (2025-10-31 21:05:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 370.48673ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
559.325598ms
FAIL (2025-10-31 21:05:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 559.341086ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
419.807765ms
FAIL (2025-10-31 21:04:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
tBegin
Oct 31 21:03:12.217456618 (105h49m38.918401503s) [upload=93134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:12.238821717 (105h49m38.939766612s) [seg_pos={0 0}, upload=93134]: Begin upload segment...
Oct 31 21:03:12.238864187 (105h49m38.939809082s) [seg_pos={0 0}, upload=93134]: Done begin upload segment.
Oct 31 21:03:12.238885798 (105h49m38.939830683s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.238936744 (105h49m38.939881628s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239432864 (105h49m38.940377749s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239474923 (105h49m38.940419818s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239689976 (105h49m38.940634861s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239937421 (105h49m38.940882305s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239946858 (105h49m38.940891753s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240172622 (105h49m38.941117517s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240219950 (105h49m38.941164835s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240415367 (105h49m38.941360262s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240640008 (105h49m38.941584893s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240714867 (105h49m38.941659752s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240874156 (105h49m38.941819041s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241069643 (105h49m38.942014538s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241168208 (105h49m38.942113103s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241718780 (105h49m38.942663665s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241812185 (105h49m38.942757081s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.244034422 (105h49m38.944979318s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.342147728 (105h49m39.043092643s) [upload=93134]: Next returned nil segment
Oct 31 21:03:12.342173847 (105h49m39.043118802s) [upload=93134]: Waiting for error group managing segments...
Oct 31 21:03:12.352785595 (105h49m39.053730511s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.352905409 (105h49m39.053850304s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.365098953 (105h49m39.066043858s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.367756977 (105h49m39.068701892s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.372646285 (105h49m39.07359121s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.374120059 (105h49m39.075064973s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.378605118 (105h49m39.079550023s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.550417419 (105h49m39.251362304s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.875337427 (105h49m39.576282322s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:37.185165129 (105h50m3.886110024s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:00.750493652 (105h50m27.451438537s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.070380024 (105h50m33.771324919s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.970385749 (105h50m34.671330644s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:12.205981337 (105h50m38.906926221s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206045126 (105h50m38.906990011s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12VfmooWvtic6UbcWtzBQ4ttYgUPbo8sjCJyM5VVCXgcQw6Ld6F, address:45.91.116.14:4205): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206074311 (105h50m38.907019196s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection
Oct 31 21:04:12.206096242 (105h50m38.907041117s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1YzWUkjkJXsP2Y5JYoFBM8Z8gsFbC5hzNGbWRgRTwg4uGge71p, address:45.91.116.6:4997): protocol: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection
Oct 31 21:04:12.206120908 (105h50m38.907065793s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: expected piece hash; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; EOF
Oct 31 21:04:12.206143972 (105h50m38.907088857s) [seg_pos={0 0}, upload=93134]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:04:12.226439497 (105h50m38.927384382s) [upload=93134]: Done uploading segments...
Oct 31 21:04:12.226460978 (105h50m38.927405873s) [upload=93134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 419.829274ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
573.980095ms
FAIL (2025-10-31 21:04:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 574.002928ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
883.56655ms
FAIL (2025-10-31 21:04:27 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 883.586878ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
577.805947ms
FAIL (2025-10-31 21:04:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
tBegin
Oct 31 21:03:12.217456618 (105h49m38.918401503s) [upload=93134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:12.238821717 (105h49m38.939766612s) [seg_pos={0 0}, upload=93134]: Begin upload segment...
Oct 31 21:03:12.238864187 (105h49m38.939809082s) [seg_pos={0 0}, upload=93134]: Done begin upload segment.
Oct 31 21:03:12.238885798 (105h49m38.939830683s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.238936744 (105h49m38.939881628s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239432864 (105h49m38.940377749s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239474923 (105h49m38.940419818s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239689976 (105h49m38.940634861s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239937421 (105h49m38.940882305s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239946858 (105h49m38.940891753s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240172622 (105h49m38.941117517s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240219950 (105h49m38.941164835s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240415367 (105h49m38.941360262s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240640008 (105h49m38.941584893s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240714867 (105h49m38.941659752s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240874156 (105h49m38.941819041s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241069643 (105h49m38.942014538s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241168208 (105h49m38.942113103s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241718780 (105h49m38.942663665s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241812185 (105h49m38.942757081s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.244034422 (105h49m38.944979318s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.342147728 (105h49m39.043092643s) [upload=93134]: Next returned nil segment
Oct 31 21:03:12.342173847 (105h49m39.043118802s) [upload=93134]: Waiting for error group managing segments...
Oct 31 21:03:12.352785595 (105h49m39.053730511s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.352905409 (105h49m39.053850304s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.365098953 (105h49m39.066043858s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.367756977 (105h49m39.068701892s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.372646285 (105h49m39.07359121s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.374120059 (105h49m39.075064973s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.378605118 (105h49m39.079550023s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.550417419 (105h49m39.251362304s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.875337427 (105h49m39.576282322s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:37.185165129 (105h50m3.886110024s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:00.750493652 (105h50m27.451438537s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.070380024 (105h50m33.771324919s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.970385749 (105h50m34.671330644s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:12.205981337 (105h50m38.906926221s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206045126 (105h50m38.906990011s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12VfmooWvtic6UbcWtzBQ4ttYgUPbo8sjCJyM5VVCXgcQw6Ld6F, address:45.91.116.14:4205): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206074311 (105h50m38.907019196s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection
Oct 31 21:04:12.206096242 (105h50m38.907041117s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1YzWUkjkJXsP2Y5JYoFBM8Z8gsFbC5hzNGbWRgRTwg4uGge71p, address:45.91.116.6:4997): protocol: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection
Oct 31 21:04:12.206120908 (105h50m38.907065793s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: expected piece hash; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; EOF
Oct 31 21:04:12.206143972 (105h50m38.907088857s) [seg_pos={0 0}, upload=93134]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:04:12.226439497 (105h50m38.927384382s) [upload=93134]: Done uploading segments...
Oct 31 21:04:12.226460978 (105h50m38.927405873s) [upload=93134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 577.82947ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X................X.] duration:
427.513328ms
FAIL (2025-10-31 21:04:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 427.539067ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.022539688s
FAIL (2025-10-31 21:04:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
tBegin
Oct 31 21:03:12.217456618 (105h49m38.918401503s) [upload=93134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:12.238821717 (105h49m38.939766612s) [seg_pos={0 0}, upload=93134]: Begin upload segment...
Oct 31 21:03:12.238864187 (105h49m38.939809082s) [seg_pos={0 0}, upload=93134]: Done begin upload segment.
Oct 31 21:03:12.238885798 (105h49m38.939830683s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.238936744 (105h49m38.939881628s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239432864 (105h49m38.940377749s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239474923 (105h49m38.940419818s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239689976 (105h49m38.940634861s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239937421 (105h49m38.940882305s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239946858 (105h49m38.940891753s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240172622 (105h49m38.941117517s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240219950 (105h49m38.941164835s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240415367 (105h49m38.941360262s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240640008 (105h49m38.941584893s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240714867 (105h49m38.941659752s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240874156 (105h49m38.941819041s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241069643 (105h49m38.942014538s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241168208 (105h49m38.942113103s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241718780 (105h49m38.942663665s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241812185 (105h49m38.942757081s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.244034422 (105h49m38.944979318s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.342147728 (105h49m39.043092643s) [upload=93134]: Next returned nil segment
Oct 31 21:03:12.342173847 (105h49m39.043118802s) [upload=93134]: Waiting for error group managing segments...
Oct 31 21:03:12.352785595 (105h49m39.053730511s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.352905409 (105h49m39.053850304s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.365098953 (105h49m39.066043858s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.367756977 (105h49m39.068701892s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.372646285 (105h49m39.07359121s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.374120059 (105h49m39.075064973s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.378605118 (105h49m39.079550023s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.550417419 (105h49m39.251362304s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.875337427 (105h49m39.576282322s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:37.185165129 (105h50m3.886110024s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:00.750493652 (105h50m27.451438537s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.070380024 (105h50m33.771324919s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.970385749 (105h50m34.671330644s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:12.205981337 (105h50m38.906926221s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206045126 (105h50m38.906990011s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12VfmooWvtic6UbcWtzBQ4ttYgUPbo8sjCJyM5VVCXgcQw6Ld6F, address:45.91.116.14:4205): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206074311 (105h50m38.907019196s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection
Oct 31 21:04:12.206096242 (105h50m38.907041117s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1YzWUkjkJXsP2Y5JYoFBM8Z8gsFbC5hzNGbWRgRTwg4uGge71p, address:45.91.116.6:4997): protocol: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection
Oct 31 21:04:12.206120908 (105h50m38.907065793s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: expected piece hash; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; EOF
Oct 31 21:04:12.206143972 (105h50m38.907088857s) [seg_pos={0 0}, upload=93134]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:04:12.226439497 (105h50m38.927384382s) [upload=93134]: Done uploading segments...
Oct 31 21:04:12.226460978 (105h50m38.927405873s) [upload=93134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.022558122s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
1.141694548s
FAIL (2025-10-31 21:04:09 UTC)
orbiter warning: context deadline exceeded
Duration: 1.141715647s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
58.231063978s
FAIL (2025-10-31 21:04:05 UTC)
orbiter warning: context deadline exceeded
Duration: 58.231082822s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
942.17224ms
FAIL (2025-10-31 21:03:57 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 942.194991ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
3.979750427s
FAIL (2025-10-31 21:03:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.979775624s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.020800238s
FAIL (2025-10-31 21:03:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.020821839s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X................X] duration:
30.002054976s
FAIL (2025-10-31 21:03:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.002075315s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
895.520185ms
FAIL (2025-10-31 21:03:32 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 895.560852ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
2.035942555s
FAIL (2025-10-31 21:03:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.035972993s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
850.961482ms
FAIL (2025-10-31 21:02:55 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 850.987171ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
549.852001ms
FAIL (2025-10-31 21:02:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 549.872511ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
629.493477ms
FAIL (2025-10-31 21:02:26 UTC)
orbiter warning: context deadline exceeded
Duration: 629.556505ms
core
uplink
us-select-1
us1-placement-20
|