Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
324.888701ms
FAIL (2025-10-31 21:15:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 324.917815ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................X................X.............X..] duration:
416.850865ms
FAIL (2025-10-31 21:15:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 416.867818ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X............X...] duration:
374.950445ms
FAIL (2025-10-31 21:15:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489220300 (105h59m54.190165196s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489559817 (105h59m54.190504703s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490016915 (105h59m54.19096181s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490431101 (105h59m54.191375996s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490921210 (105h59m54.191866105s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491320298 (105h59m54.192265184s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491637283 (105h59m54.192582168s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491993701 (105h59m54.192938596s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492347455 (105h59m54.19329234s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492684076 (105h59m54.193628971s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.494314954 (105h59m54.195259849s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.576610724 (105h59m54.27755561s) [upload=93273]: Next returned nil segment
Oct 31 21:13:27.576639588 (105h59m54.277584464s) [upload=93273]: Waiting for error group managing segments...
Oct 31 21:13:27.601581249 (105h59m54.302526154s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.604395536 (105h59m54.305340441s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.613295836 (105h59m54.314240721s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.619255753 (105h59m54.320200648s) [noise=true , address=145.79.227.4:4211 , piece_id=BGTGHX22JS7DKYEUCCRTCT6QRUSCZ3RHP5KBZ3D2WTHH5E3OIKLQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.621122162 (105h59m54.322067057s) [noise=true , address=145.79.227.17:4980 , piece_id=YFU2APCK2H34O47J6XNMWTLLN5WIZIU4ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.633764760 (105h59m54.334709674s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:30.536247637 (105h59m57.237192521s) [noise=true , address=145.79.227.16:4928 , piece_id=IOUDHEZTRV6WVGXPSXWM7H4QZQTTYLWCTJKZ7B4UWARCUPR7LEJA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:14:27.438813050 (106h0m54.139757945s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4, address:45.91.116.5:4308): protocol: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection
Oct 31 21:14:27.438927465 (106h0m54.13987235s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.438957591 (106h0m54.139902476s) [noise=true , address=77.90.31.16:4750 , piece_id=JLCX7CATMWDOI2AGTLAKWG3DEXILT6MZQOY5MDSHTS62ID2C2Z5Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 5.161.89.252:50622->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439074560 (106h0m54.140019455s) [noise=true , address=45.91.116.10:4091 , piece_id=7I45ZOZE5VZB3U4KS4T7P2JPK23SSROZADKZY45PSYNSMESNPUHQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.439151815 (106h0m54.140096701s) [noise=true , address=77.90.31.10:4552 , piece_id=ZDOPX7KZYCR3TJHEIREPXRKPRD53OC65LFOIVD7PXCMGCG7XXBXA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection
Oct 31 21:14:27.439251072 (106h0m54.140195977s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection
Oct 31 21:14:27.439274105 (106h0m54.140218989s) [noise=true , address=45.91.116.8:4059 , piece_id=H2P3IEUEAWRIRW7U3BZIQJKXO7FKBSXCYIQAIPRF2ZWDPVL4JAQQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YzYcwyVJqZywPJt9tYXXHMrjfjeziu6fobhHjt6sRf3utaror, address:45.91.116.8:4059): protocol: rpc: dial tcp 45.91.116.8:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439365316 (106h0m54.140310201s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection
Oct 31 21:14:27.439469882 (106h0m54.140414787s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 5.161.89.252:52750->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439479670 (106h0m54.140424555s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: read tcp 5.161.89.252:52818->77.90.31.13:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439557907 (106h0m54.140502802s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: read tcp 5.161.89.252:31250->77.90.31.19:4868: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439574608 (106h0m54.140519503s) [seg_pos={0 0}, upload=93273]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Oct 31 21:14:27.456689141 (106h0m54.157634026s) [upload=93273]: Done uploading segments...
Oct 31 21:14:27.456707465 (106h0m54.15765236s) [upload=93273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 374.971535ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
715.928365ms
FAIL (2025-10-31 21:15:38 UTC)
orbiter warning: context deadline exceeded
Duration: 715.945246ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [.......................X.................................X..] duration:
611.349227ms
FAIL (2025-10-31 21:15:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 611.365657ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X............X..] duration:
384.128226ms
FAIL (2025-10-31 21:15:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489220300 (105h59m54.190165196s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489559817 (105h59m54.190504703s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490016915 (105h59m54.19096181s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490431101 (105h59m54.191375996s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490921210 (105h59m54.191866105s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491320298 (105h59m54.192265184s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491637283 (105h59m54.192582168s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491993701 (105h59m54.192938596s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492347455 (105h59m54.19329234s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492684076 (105h59m54.193628971s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.494314954 (105h59m54.195259849s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.576610724 (105h59m54.27755561s) [upload=93273]: Next returned nil segment
Oct 31 21:13:27.576639588 (105h59m54.277584464s) [upload=93273]: Waiting for error group managing segments...
Oct 31 21:13:27.601581249 (105h59m54.302526154s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.604395536 (105h59m54.305340441s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.613295836 (105h59m54.314240721s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.619255753 (105h59m54.320200648s) [noise=true , address=145.79.227.4:4211 , piece_id=BGTGHX22JS7DKYEUCCRTCT6QRUSCZ3RHP5KBZ3D2WTHH5E3OIKLQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.621122162 (105h59m54.322067057s) [noise=true , address=145.79.227.17:4980 , piece_id=YFU2APCK2H34O47J6XNMWTLLN5WIZIU4ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.633764760 (105h59m54.334709674s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:30.536247637 (105h59m57.237192521s) [noise=true , address=145.79.227.16:4928 , piece_id=IOUDHEZTRV6WVGXPSXWM7H4QZQTTYLWCTJKZ7B4UWARCUPR7LEJA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:14:27.438813050 (106h0m54.139757945s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4, address:45.91.116.5:4308): protocol: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection
Oct 31 21:14:27.438927465 (106h0m54.13987235s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.438957591 (106h0m54.139902476s) [noise=true , address=77.90.31.16:4750 , piece_id=JLCX7CATMWDOI2AGTLAKWG3DEXILT6MZQOY5MDSHTS62ID2C2Z5Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 5.161.89.252:50622->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439074560 (106h0m54.140019455s) [noise=true , address=45.91.116.10:4091 , piece_id=7I45ZOZE5VZB3U4KS4T7P2JPK23SSROZADKZY45PSYNSMESNPUHQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.439151815 (106h0m54.140096701s) [noise=true , address=77.90.31.10:4552 , piece_id=ZDOPX7KZYCR3TJHEIREPXRKPRD53OC65LFOIVD7PXCMGCG7XXBXA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection
Oct 31 21:14:27.439251072 (106h0m54.140195977s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection
Oct 31 21:14:27.439274105 (106h0m54.140218989s) [noise=true , address=45.91.116.8:4059 , piece_id=H2P3IEUEAWRIRW7U3BZIQJKXO7FKBSXCYIQAIPRF2ZWDPVL4JAQQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YzYcwyVJqZywPJt9tYXXHMrjfjeziu6fobhHjt6sRf3utaror, address:45.91.116.8:4059): protocol: rpc: dial tcp 45.91.116.8:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439365316 (106h0m54.140310201s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection
Oct 31 21:14:27.439469882 (106h0m54.140414787s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 5.161.89.252:52750->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439479670 (106h0m54.140424555s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: read tcp 5.161.89.252:52818->77.90.31.13:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439557907 (106h0m54.140502802s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: read tcp 5.161.89.252:31250->77.90.31.19:4868: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439574608 (106h0m54.140519503s) [seg_pos={0 0}, upload=93273]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Oct 31 21:14:27.456689141 (106h0m54.157634026s) [upload=93273]: Done uploading segments...
Oct 31 21:14:27.456707465 (106h0m54.15765236s) [upload=93273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 384.150518ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
763.565724ms
FAIL (2025-10-31 21:14:50 UTC)
orbiter warning: context deadline exceeded
Duration: 763.581083ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
509.444977ms
FAIL (2025-10-31 21:14:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 509.466588ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................X................X.............X.] duration:
1.466310377s
FAIL (2025-10-31 21:14:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.466335974s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
28.929740818s
FAIL (2025-10-31 21:14:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 28.929756909s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
1.113753133s
FAIL (2025-10-31 21:14:34 UTC)
orbiter warning: context deadline exceeded
Duration: 1.113769794s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X............X.] duration:
1.855098778s
FAIL (2025-10-31 21:14:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489220300 (105h59m54.190165196s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489559817 (105h59m54.190504703s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490016915 (105h59m54.19096181s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490431101 (105h59m54.191375996s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490921210 (105h59m54.191866105s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491320298 (105h59m54.192265184s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491637283 (105h59m54.192582168s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491993701 (105h59m54.192938596s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492347455 (105h59m54.19329234s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492684076 (105h59m54.193628971s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.494314954 (105h59m54.195259849s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.576610724 (105h59m54.27755561s) [upload=93273]: Next returned nil segment
Oct 31 21:13:27.576639588 (105h59m54.277584464s) [upload=93273]: Waiting for error group managing segments...
Oct 31 21:13:27.601581249 (105h59m54.302526154s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.604395536 (105h59m54.305340441s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.613295836 (105h59m54.314240721s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.619255753 (105h59m54.320200648s) [noise=true , address=145.79.227.4:4211 , piece_id=BGTGHX22JS7DKYEUCCRTCT6QRUSCZ3RHP5KBZ3D2WTHH5E3OIKLQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.621122162 (105h59m54.322067057s) [noise=true , address=145.79.227.17:4980 , piece_id=YFU2APCK2H34O47J6XNMWTLLN5WIZIU4ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.633764760 (105h59m54.334709674s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:30.536247637 (105h59m57.237192521s) [noise=true , address=145.79.227.16:4928 , piece_id=IOUDHEZTRV6WVGXPSXWM7H4QZQTTYLWCTJKZ7B4UWARCUPR7LEJA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:14:27.438813050 (106h0m54.139757945s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4, address:45.91.116.5:4308): protocol: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection
Oct 31 21:14:27.438927465 (106h0m54.13987235s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.438957591 (106h0m54.139902476s) [noise=true , address=77.90.31.16:4750 , piece_id=JLCX7CATMWDOI2AGTLAKWG3DEXILT6MZQOY5MDSHTS62ID2C2Z5Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 5.161.89.252:50622->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439074560 (106h0m54.140019455s) [noise=true , address=45.91.116.10:4091 , piece_id=7I45ZOZE5VZB3U4KS4T7P2JPK23SSROZADKZY45PSYNSMESNPUHQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.439151815 (106h0m54.140096701s) [noise=true , address=77.90.31.10:4552 , piece_id=ZDOPX7KZYCR3TJHEIREPXRKPRD53OC65LFOIVD7PXCMGCG7XXBXA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection
Oct 31 21:14:27.439251072 (106h0m54.140195977s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection
Oct 31 21:14:27.439274105 (106h0m54.140218989s) [noise=true , address=45.91.116.8:4059 , piece_id=H2P3IEUEAWRIRW7U3BZIQJKXO7FKBSXCYIQAIPRF2ZWDPVL4JAQQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YzYcwyVJqZywPJt9tYXXHMrjfjeziu6fobhHjt6sRf3utaror, address:45.91.116.8:4059): protocol: rpc: dial tcp 45.91.116.8:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439365316 (106h0m54.140310201s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection
Oct 31 21:14:27.439469882 (106h0m54.140414787s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 5.161.89.252:52750->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439479670 (106h0m54.140424555s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: read tcp 5.161.89.252:52818->77.90.31.13:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439557907 (106h0m54.140502802s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: read tcp 5.161.89.252:31250->77.90.31.19:4868: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439574608 (106h0m54.140519503s) [seg_pos={0 0}, upload=93273]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Oct 31 21:14:27.456689141 (106h0m54.157634026s) [upload=93273]: Done uploading segments...
Oct 31 21:14:27.456707465 (106h0m54.15765236s) [upload=93273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 1.855114368s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X............X] duration:
1m0.01908996s
FAIL (2025-10-31 21:14:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489220300 (105h59m54.190165196s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489559817 (105h59m54.190504703s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490016915 (105h59m54.19096181s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490431101 (105h59m54.191375996s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490921210 (105h59m54.191866105s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491320298 (105h59m54.192265184s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491637283 (105h59m54.192582168s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491993701 (105h59m54.192938596s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492347455 (105h59m54.19329234s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492684076 (105h59m54.193628971s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.494314954 (105h59m54.195259849s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.576610724 (105h59m54.27755561s) [upload=93273]: Next returned nil segment
Oct 31 21:13:27.576639588 (105h59m54.277584464s) [upload=93273]: Waiting for error group managing segments...
Oct 31 21:13:27.601581249 (105h59m54.302526154s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.604395536 (105h59m54.305340441s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.613295836 (105h59m54.314240721s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.619255753 (105h59m54.320200648s) [noise=true , address=145.79.227.4:4211 , piece_id=BGTGHX22JS7DKYEUCCRTCT6QRUSCZ3RHP5KBZ3D2WTHH5E3OIKLQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.621122162 (105h59m54.322067057s) [noise=true , address=145.79.227.17:4980 , piece_id=YFU2APCK2H34O47J6XNMWTLLN5WIZIU4ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.633764760 (105h59m54.334709674s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:30.536247637 (105h59m57.237192521s) [noise=true , address=145.79.227.16:4928 , piece_id=IOUDHEZTRV6WVGXPSXWM7H4QZQTTYLWCTJKZ7B4UWARCUPR7LEJA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:14:27.438813050 (106h0m54.139757945s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4, address:45.91.116.5:4308): protocol: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection
Oct 31 21:14:27.438927465 (106h0m54.13987235s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.438957591 (106h0m54.139902476s) [noise=true , address=77.90.31.16:4750 , piece_id=JLCX7CATMWDOI2AGTLAKWG3DEXILT6MZQOY5MDSHTS62ID2C2Z5Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 5.161.89.252:50622->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439074560 (106h0m54.140019455s) [noise=true , address=45.91.116.10:4091 , piece_id=7I45ZOZE5VZB3U4KS4T7P2JPK23SSROZADKZY45PSYNSMESNPUHQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.439151815 (106h0m54.140096701s) [noise=true , address=77.90.31.10:4552 , piece_id=ZDOPX7KZYCR3TJHEIREPXRKPRD53OC65LFOIVD7PXCMGCG7XXBXA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection
Oct 31 21:14:27.439251072 (106h0m54.140195977s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection
Oct 31 21:14:27.439274105 (106h0m54.140218989s) [noise=true , address=45.91.116.8:4059 , piece_id=H2P3IEUEAWRIRW7U3BZIQJKXO7FKBSXCYIQAIPRF2ZWDPVL4JAQQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YzYcwyVJqZywPJt9tYXXHMrjfjeziu6fobhHjt6sRf3utaror, address:45.91.116.8:4059): protocol: rpc: dial tcp 45.91.116.8:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439365316 (106h0m54.140310201s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection
Oct 31 21:14:27.439469882 (106h0m54.140414787s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 5.161.89.252:52750->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439479670 (106h0m54.140424555s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: read tcp 5.161.89.252:52818->77.90.31.13:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439557907 (106h0m54.140502802s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: read tcp 5.161.89.252:31250->77.90.31.19:4868: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439574608 (106h0m54.140519503s) [seg_pos={0 0}, upload=93273]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Oct 31 21:14:27.456689141 (106h0m54.157634026s) [upload=93273]: Done uploading segments...
Oct 31 21:14:27.456707465 (106h0m54.15765236s) [upload=93273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 1m0.019104077s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [........................X.................................X.] duration:
2.556984762s
FAIL (2025-10-31 21:14:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.557002335s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................X................X.............X] duration:
30.001177665s
FAIL (2025-10-31 21:14:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001195447s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [.........................X.................................X] duration:
5.000683665s
FAIL (2025-10-31 21:13:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000701731s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
821.683986ms
FAIL (2025-10-31 21:13:24 UTC)
orbiter warning: context deadline exceeded
Duration: 821.701419ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X................X.............] duration:
334.600972ms
FAIL (2025-10-31 21:13:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 334.619196ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
845.229879ms
FAIL (2025-10-31 21:13:05 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 845.250547ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
440.009343ms
FAIL (2025-10-31 21:12:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 440.035612ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X................X............] duration:
369.155227ms
FAIL (2025-10-31 21:12:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 369.171577ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
551.852652ms
FAIL (2025-10-31 21:12:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 552.054139ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
441.944832ms
FAIL (2025-10-31 21:12:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
tBegin
Oct 31 21:03:12.217456618 (105h49m38.918401503s) [upload=93134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:12.238821717 (105h49m38.939766612s) [seg_pos={0 0}, upload=93134]: Begin upload segment...
Oct 31 21:03:12.238864187 (105h49m38.939809082s) [seg_pos={0 0}, upload=93134]: Done begin upload segment.
Oct 31 21:03:12.238885798 (105h49m38.939830683s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.238936744 (105h49m38.939881628s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239432864 (105h49m38.940377749s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239474923 (105h49m38.940419818s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239689976 (105h49m38.940634861s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239937421 (105h49m38.940882305s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239946858 (105h49m38.940891753s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240172622 (105h49m38.941117517s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240219950 (105h49m38.941164835s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240415367 (105h49m38.941360262s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240640008 (105h49m38.941584893s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240714867 (105h49m38.941659752s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240874156 (105h49m38.941819041s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241069643 (105h49m38.942014538s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241168208 (105h49m38.942113103s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241718780 (105h49m38.942663665s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241812185 (105h49m38.942757081s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.244034422 (105h49m38.944979318s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.342147728 (105h49m39.043092643s) [upload=93134]: Next returned nil segment
Oct 31 21:03:12.342173847 (105h49m39.043118802s) [upload=93134]: Waiting for error group managing segments...
Oct 31 21:03:12.352785595 (105h49m39.053730511s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.352905409 (105h49m39.053850304s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.365098953 (105h49m39.066043858s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.367756977 (105h49m39.068701892s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.372646285 (105h49m39.07359121s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.374120059 (105h49m39.075064973s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.378605118 (105h49m39.079550023s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.550417419 (105h49m39.251362304s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.875337427 (105h49m39.576282322s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:37.185165129 (105h50m3.886110024s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:00.750493652 (105h50m27.451438537s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.070380024 (105h50m33.771324919s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.970385749 (105h50m34.671330644s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:12.205981337 (105h50m38.906926221s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206045126 (105h50m38.906990011s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12VfmooWvtic6UbcWtzBQ4ttYgUPbo8sjCJyM5VVCXgcQw6Ld6F, address:45.91.116.14:4205): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206074311 (105h50m38.907019196s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection
Oct 31 21:04:12.206096242 (105h50m38.907041117s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1YzWUkjkJXsP2Y5JYoFBM8Z8gsFbC5hzNGbWRgRTwg4uGge71p, address:45.91.116.6:4997): protocol: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection
Oct 31 21:04:12.206120908 (105h50m38.907065793s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: expected piece hash; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; EOF
Oct 31 21:04:12.206143972 (105h50m38.907088857s) [seg_pos={0 0}, upload=93134]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:04:12.226439497 (105h50m38.927384382s) [upload=93134]: Done uploading segments...
Oct 31 21:04:12.226460978 (105h50m38.927405873s) [upload=93134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 441.969027ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
921.663466ms
FAIL (2025-10-31 21:12:41 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 921.713309ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
455.714773ms
FAIL (2025-10-31 21:12:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
tBegin
Oct 31 21:03:12.217456618 (105h49m38.918401503s) [upload=93134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:12.238821717 (105h49m38.939766612s) [seg_pos={0 0}, upload=93134]: Begin upload segment...
Oct 31 21:03:12.238864187 (105h49m38.939809082s) [seg_pos={0 0}, upload=93134]: Done begin upload segment.
Oct 31 21:03:12.238885798 (105h49m38.939830683s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.238936744 (105h49m38.939881628s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239432864 (105h49m38.940377749s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239474923 (105h49m38.940419818s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239689976 (105h49m38.940634861s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239937421 (105h49m38.940882305s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239946858 (105h49m38.940891753s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240172622 (105h49m38.941117517s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240219950 (105h49m38.941164835s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240415367 (105h49m38.941360262s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240640008 (105h49m38.941584893s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240714867 (105h49m38.941659752s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240874156 (105h49m38.941819041s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241069643 (105h49m38.942014538s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241168208 (105h49m38.942113103s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241718780 (105h49m38.942663665s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241812185 (105h49m38.942757081s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.244034422 (105h49m38.944979318s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.342147728 (105h49m39.043092643s) [upload=93134]: Next returned nil segment
Oct 31 21:03:12.342173847 (105h49m39.043118802s) [upload=93134]: Waiting for error group managing segments...
Oct 31 21:03:12.352785595 (105h49m39.053730511s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.352905409 (105h49m39.053850304s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.365098953 (105h49m39.066043858s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.367756977 (105h49m39.068701892s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.372646285 (105h49m39.07359121s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.374120059 (105h49m39.075064973s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.378605118 (105h49m39.079550023s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.550417419 (105h49m39.251362304s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.875337427 (105h49m39.576282322s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:37.185165129 (105h50m3.886110024s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:00.750493652 (105h50m27.451438537s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.070380024 (105h50m33.771324919s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.970385749 (105h50m34.671330644s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:12.205981337 (105h50m38.906926221s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206045126 (105h50m38.906990011s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12VfmooWvtic6UbcWtzBQ4ttYgUPbo8sjCJyM5VVCXgcQw6Ld6F, address:45.91.116.14:4205): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206074311 (105h50m38.907019196s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection
Oct 31 21:04:12.206096242 (105h50m38.907041117s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1YzWUkjkJXsP2Y5JYoFBM8Z8gsFbC5hzNGbWRgRTwg4uGge71p, address:45.91.116.6:4997): protocol: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection
Oct 31 21:04:12.206120908 (105h50m38.907065793s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: expected piece hash; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; EOF
Oct 31 21:04:12.206143972 (105h50m38.907088857s) [seg_pos={0 0}, upload=93134]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:04:12.226439497 (105h50m38.927384382s) [upload=93134]: Done uploading segments...
Oct 31 21:04:12.226460978 (105h50m38.927405873s) [upload=93134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 455.736344ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
638.513356ms
FAIL (2025-10-31 21:12:16 UTC)
orbiter warning: context deadline exceeded
Duration: 638.720844ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X................X...........] duration:
348.612126ms
FAIL (2025-10-31 21:12:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 348.628526ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
392.282965ms
FAIL (2025-10-31 21:11:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 392.304746ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
687.496147ms
FAIL (2025-10-31 21:11:42 UTC)
orbiter warning: context deadline exceeded
Duration: 687.511867ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
825.463444ms
FAIL (2025-10-31 21:11:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 825.484073ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
917.040127ms
FAIL (2025-10-31 21:11:35 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 917.058371ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................X................X..........] duration:
456.606001ms
FAIL (2025-10-31 21:11:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 456.626278ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
550.223346ms
FAIL (2025-10-31 21:11:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
tBegin
Oct 31 21:03:12.217456618 (105h49m38.918401503s) [upload=93134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:12.238821717 (105h49m38.939766612s) [seg_pos={0 0}, upload=93134]: Begin upload segment...
Oct 31 21:03:12.238864187 (105h49m38.939809082s) [seg_pos={0 0}, upload=93134]: Done begin upload segment.
Oct 31 21:03:12.238885798 (105h49m38.939830683s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.238936744 (105h49m38.939881628s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239432864 (105h49m38.940377749s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239474923 (105h49m38.940419818s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239689976 (105h49m38.940634861s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239937421 (105h49m38.940882305s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239946858 (105h49m38.940891753s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240172622 (105h49m38.941117517s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240219950 (105h49m38.941164835s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240415367 (105h49m38.941360262s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240640008 (105h49m38.941584893s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240714867 (105h49m38.941659752s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240874156 (105h49m38.941819041s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241069643 (105h49m38.942014538s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241168208 (105h49m38.942113103s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241718780 (105h49m38.942663665s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241812185 (105h49m38.942757081s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.244034422 (105h49m38.944979318s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.342147728 (105h49m39.043092643s) [upload=93134]: Next returned nil segment
Oct 31 21:03:12.342173847 (105h49m39.043118802s) [upload=93134]: Waiting for error group managing segments...
Oct 31 21:03:12.352785595 (105h49m39.053730511s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.352905409 (105h49m39.053850304s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.365098953 (105h49m39.066043858s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.367756977 (105h49m39.068701892s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.372646285 (105h49m39.07359121s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.374120059 (105h49m39.075064973s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.378605118 (105h49m39.079550023s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.550417419 (105h49m39.251362304s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.875337427 (105h49m39.576282322s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:37.185165129 (105h50m3.886110024s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:00.750493652 (105h50m27.451438537s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.070380024 (105h50m33.771324919s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.970385749 (105h50m34.671330644s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:12.205981337 (105h50m38.906926221s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206045126 (105h50m38.906990011s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12VfmooWvtic6UbcWtzBQ4ttYgUPbo8sjCJyM5VVCXgcQw6Ld6F, address:45.91.116.14:4205): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206074311 (105h50m38.907019196s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection
Oct 31 21:04:12.206096242 (105h50m38.907041117s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1YzWUkjkJXsP2Y5JYoFBM8Z8gsFbC5hzNGbWRgRTwg4uGge71p, address:45.91.116.6:4997): protocol: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection
Oct 31 21:04:12.206120908 (105h50m38.907065793s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: expected piece hash; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; EOF
Oct 31 21:04:12.206143972 (105h50m38.907088857s) [seg_pos={0 0}, upload=93134]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:04:12.226439497 (105h50m38.927384382s) [upload=93134]: Done uploading segments...
Oct 31 21:04:12.226460978 (105h50m38.927405873s) [upload=93134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 550.249024ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
496.201422ms
FAIL (2025-10-31 21:11:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 496.220749ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
420.944119ms
FAIL (2025-10-31 21:10:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 420.963815ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
867.345787ms
FAIL (2025-10-31 21:10:56 UTC)
orbiter warning: context deadline exceeded
Duration: 867.365884ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
496.485845ms
FAIL (2025-10-31 21:10:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 496.504689ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
919.248246ms
FAIL (2025-10-31 21:10:36 UTC)
orbiter warning: context deadline exceeded
Duration: 919.264997ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
841.914791ms
FAIL (2025-10-31 21:10:34 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 841.93496ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
406.595365ms
FAIL (2025-10-31 21:10:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
tBegin
Oct 31 21:03:12.217456618 (105h49m38.918401503s) [upload=93134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:12.238821717 (105h49m38.939766612s) [seg_pos={0 0}, upload=93134]: Begin upload segment...
Oct 31 21:03:12.238864187 (105h49m38.939809082s) [seg_pos={0 0}, upload=93134]: Done begin upload segment.
Oct 31 21:03:12.238885798 (105h49m38.939830683s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.238936744 (105h49m38.939881628s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239432864 (105h49m38.940377749s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239474923 (105h49m38.940419818s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239689976 (105h49m38.940634861s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239937421 (105h49m38.940882305s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239946858 (105h49m38.940891753s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240172622 (105h49m38.941117517s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240219950 (105h49m38.941164835s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240415367 (105h49m38.941360262s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240640008 (105h49m38.941584893s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240714867 (105h49m38.941659752s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240874156 (105h49m38.941819041s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241069643 (105h49m38.942014538s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241168208 (105h49m38.942113103s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241718780 (105h49m38.942663665s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241812185 (105h49m38.942757081s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.244034422 (105h49m38.944979318s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.342147728 (105h49m39.043092643s) [upload=93134]: Next returned nil segment
Oct 31 21:03:12.342173847 (105h49m39.043118802s) [upload=93134]: Waiting for error group managing segments...
Oct 31 21:03:12.352785595 (105h49m39.053730511s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.352905409 (105h49m39.053850304s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.365098953 (105h49m39.066043858s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.367756977 (105h49m39.068701892s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.372646285 (105h49m39.07359121s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.374120059 (105h49m39.075064973s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.378605118 (105h49m39.079550023s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.550417419 (105h49m39.251362304s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.875337427 (105h49m39.576282322s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:37.185165129 (105h50m3.886110024s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:00.750493652 (105h50m27.451438537s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.070380024 (105h50m33.771324919s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.970385749 (105h50m34.671330644s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:12.205981337 (105h50m38.906926221s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206045126 (105h50m38.906990011s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12VfmooWvtic6UbcWtzBQ4ttYgUPbo8sjCJyM5VVCXgcQw6Ld6F, address:45.91.116.14:4205): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206074311 (105h50m38.907019196s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection
Oct 31 21:04:12.206096242 (105h50m38.907041117s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1YzWUkjkJXsP2Y5JYoFBM8Z8gsFbC5hzNGbWRgRTwg4uGge71p, address:45.91.116.6:4997): protocol: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection
Oct 31 21:04:12.206120908 (105h50m38.907065793s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: expected piece hash; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; EOF
Oct 31 21:04:12.206143972 (105h50m38.907088857s) [seg_pos={0 0}, upload=93134]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:04:12.226439497 (105h50m38.927384382s) [upload=93134]: Done uploading segments...
Oct 31 21:04:12.226460978 (105h50m38.927405873s) [upload=93134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 406.618187ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X................X.........] duration:
351.3091ms
FAIL (2025-10-31 21:10:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 351.329048ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
879.725619ms
FAIL (2025-10-31 21:10:07 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 879.754213ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
386.950338ms
FAIL (2025-10-31 21:09:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 386.968492ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
762.771979ms
FAIL (2025-10-31 21:09:36 UTC)
orbiter warning: context deadline exceeded
Duration: 762.793278ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
513.387806ms
FAIL (2025-10-31 21:09:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 513.404046ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................X................X........] duration:
366.976156ms
FAIL (2025-10-31 21:09:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 366.994972ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
360.308112ms
FAIL (2025-10-31 21:09:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
tBegin
Oct 31 21:03:12.217456618 (105h49m38.918401503s) [upload=93134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:12.238821717 (105h49m38.939766612s) [seg_pos={0 0}, upload=93134]: Begin upload segment...
Oct 31 21:03:12.238864187 (105h49m38.939809082s) [seg_pos={0 0}, upload=93134]: Done begin upload segment.
Oct 31 21:03:12.238885798 (105h49m38.939830683s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.238936744 (105h49m38.939881628s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239432864 (105h49m38.940377749s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239474923 (105h49m38.940419818s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239689976 (105h49m38.940634861s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239937421 (105h49m38.940882305s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.239946858 (105h49m38.940891753s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240172622 (105h49m38.941117517s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240219950 (105h49m38.941164835s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240415367 (105h49m38.941360262s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240640008 (105h49m38.941584893s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240714867 (105h49m38.941659752s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.240874156 (105h49m38.941819041s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241069643 (105h49m38.942014538s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241168208 (105h49m38.942113103s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241718780 (105h49m38.942663665s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.241812185 (105h49m38.942757081s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.244034422 (105h49m38.944979318s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Uploading piece...
Oct 31 21:03:12.342147728 (105h49m39.043092643s) [upload=93134]: Next returned nil segment
Oct 31 21:03:12.342173847 (105h49m39.043118802s) [upload=93134]: Waiting for error group managing segments...
Oct 31 21:03:12.352785595 (105h49m39.053730511s) [noise=true , address=145.79.227.10:4411 , piece_id=6T5KD4MMNY2MSVEQ63ZWTHIZONIZSHPBSCU2ZE5QZ3BRTORJR64Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.352905409 (105h49m39.053850304s) [noise=true , address=145.79.227.3:4168 , piece_id=63VIZ5S7AEYS6SO3NRK46TLCGKZPI4YUCRIKI2ERMOU77YMR36EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.365098953 (105h49m39.066043858s) [noise=true , address=145.79.227.33:4898 , piece_id=YJU5PXDV3FGFSVYMY67C2FLD35C2JO65M6DSV3TIKAV632LN5ZBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.367756977 (105h49m39.068701892s) [noise=true , address=145.79.227.42:4124 , piece_id=JWQGNXPMC2VM5UT62OL6RM4LOLOQSJ3PLPNWALCDLDCM4CPGPPVA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.372646285 (105h49m39.07359121s) [noise=true , address=145.79.227.50:4395 , piece_id=XQY7RXSQBK5VG6DFX2JODSUSJANQPIUHWVCCQMNASBKQN3VZMI5Q, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.374120059 (105h49m39.075064973s) [noise=true , address=145.79.227.13:4820 , piece_id=46TI6TOJU5QXFPWL6ELZVAAGJCXLNGWFQHNRNK4ISJ26427XISBA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.378605118 (105h49m39.079550023s) [noise=true , address=145.79.227.12:4800 , piece_id=W7P4KI22EIVIWSTT77AKFS2J6IZ4TOVU7DM35CVWGAQ4RI2K77EA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.550417419 (105h49m39.251362304s) [noise=true , address=145.79.227.49:4341 , piece_id=QUK6RWMWB7MV27YDL344QV4DL76KLUE5ODIJDKIXLNB5N6X4MZSA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:12.875337427 (105h49m39.576282322s) [noise=true , address=64.49.13.140:4514 , piece_id=LIJMTDXQSCKGZFIZ7BRMNGS3C3JM2HNWXFA3YATDY7BCJPJM6ONA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:03:37.185165129 (105h50m3.886110024s) [noise=true , address=45.91.116.12:4254 , piece_id=PM4QRDQK3YKIAJ7XDWFODKTONENPIUMLF6C7HX6BV3DSJTXVSG7A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:00.750493652 (105h50m27.451438537s) [noise=true , address=77.90.31.16:4738 , piece_id=VI4DV67WMEQYVWSNWTVMD7HVEZ7Z42IY3JSKCFVJDZCPNG7KVY5A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.070380024 (105h50m33.771324919s) [noise=true , address=77.90.31.14:4688 , piece_id=GVTLWI462A6LFK5XMMTJDZ3ID6OFKUTMIE5KP6FCOLKORYP4EVTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:07.970385749 (105h50m34.671330644s) [noise=true , address=45.91.116.4:4960 , piece_id=VGOC5BS6ABCVW6P7RRAUI7WMA4GQLJ2KH3ZF2IWQKHMNUAWWQGTA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: <nil>
Oct 31 21:04:12.205981337 (105h50m38.906926221s) [noise=true , address=77.90.31.11:4571 , piece_id=7BNVJPGPHHCBPEOLEW2S6CQFFSENVPI3OZOKVE3T6JRIUMS5KABA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206045126 (105h50m38.906990011s) [noise=true , address=45.91.116.14:4205 , piece_id=Z7CELPPM5KU2FWTYV3O7OZDLNZTGQLL3TW77BZIJS3Q4K2WSEXDQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:12VfmooWvtic6UbcWtzBQ4ttYgUPbo8sjCJyM5VVCXgcQw6Ld6F, address:45.91.116.14:4205): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:04:12.206074311 (105h50m38.907019196s) [noise=true , address=77.90.31.12:4595 , piece_id=FAETZIGN2XVNAMI43JW7SFI64HY4K4JJDT3KC5EDFMYJFSJ5FUEQ, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34462->77.90.31.12:4595: use of closed network connection
Oct 31 21:04:12.206096242 (105h50m38.907041117s) [noise=true , address=45.91.116.6:4997 , piece_id=72MYENXLDFPA2JHF3DEXWPBYS54LW4IJDDQVUYPOYHZN4DNTPH6A, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1YzWUkjkJXsP2Y5JYoFBM8Z8gsFbC5hzNGbWRgRTwg4uGge71p, address:45.91.116.6:4997): protocol: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:34260->45.91.116.6:4997: use of closed network connection
Oct 31 21:04:12.206120908 (105h50m38.907065793s) [noise=true , address=77.90.31.15:4723 , piece_id=FOEJ2ERTPCHSSJBH3JF4JROMI2VY74CNSP7BVFDS3WERPRAXRRKA, seg_pos={0 0}, upload=93134]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: expected piece hash; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; write tcp 5.161.89.252:60506->77.90.31.15:4723: use of closed network connection; EOF
Oct 31 21:04:12.206143972 (105h50m38.907088857s) [seg_pos={0 0}, upload=93134]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:04:12.226439497 (105h50m38.927384382s) [upload=93134]: Done uploading segments...
Oct 31 21:04:12.226460978 (105h50m38.927405873s) [upload=93134]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 360.329081ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
597.138867ms
FAIL (2025-10-31 21:09:06 UTC)
orbiter warning: context deadline exceeded
Duration: 597.157881ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
400.01597ms
FAIL (2025-10-31 21:08:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 400.039413ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
876.613654ms
FAIL (2025-10-31 21:08:54 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 876.6381ms
core
uplink
eu1
|