Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 375.733113ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:19:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 375.743092ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 491.250137ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:19:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 491.276818ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 15 out of 60 is failed (threshold=3) [...X...XX.....XX..X...X.....X.X.......X...X.....X.X.X...X...] duration: 7.996135329s trace: https://jaeger.dev.storj.io/trace/14a5491891bee05e FAIL (2025-12-09 01:19:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== address=185.167.204.205:4001 , piece_id=VY7CIY72VGBGZSFEI2H43N4ELABFIXCICEVQNEZB5TWRM2WMZSEQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.584421346 (81h46m58.281989662s) [noise=true , address=89.149.201.147:4008 , piece_id=HVYVZARPZ5WM3ISOGULFNA6WPCKKC3PRIMCE7226JTYCPDVXMUBA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.584711679 (81h46m58.282279995s) [noise=true , address=81.171.14.70:4034 , piece_id=GHSKMR2MYLHMKRQBNZMWHTOICWQIRRJLYVQQISQOGMT45PF7SSSQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585013122 (81h46m58.282581427s) [noise=true , address=81.171.14.70:4045 , piece_id=FD2J6MOI2FZWTDD4JCGAEQHGK6AUQNEQWLQ7PRGCJIV7L4VEXWDQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585303003 (81h46m58.282871309s) [noise=true , address=81.171.14.70:4025 , piece_id=C7OUWVW2X67WYCR7HWPAFOQO7EHTMURUNYS4JC65C3LK4YSD4MAQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585619755 (81h46m58.283188071s) [noise=true , address=185.167.204.206:4071 , piece_id=KA4HORF25UFNNH4GWCQQUSWVJCSHDBLVR3K6I77AQ7UW2AUSSBRQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585940374 (81h46m58.283508679s) [noise=true , address=185.167.204.206:4043 , piece_id=A35E7GZE4ATEJB2BCNV76WFSZV6XFCPVLD2PM3Z5H37NYUM5FDXQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.586287633 (81h46m58.283855949s) [noise=true , address=89.149.201.147:4016 , piece_id=CI3OOTGSMU2BU5EVPBPQRVAI2B6OTPY47SVW4FLV5LFT37ZAIQLA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.586579107 (81h46m58.284147423s) [noise=true , address=185.167.204.205:4036 , piece_id=2N4ZAUATTVLIRJAXGL34KBKQTN4VDEBAPTX7HK4G3IQLNU7KEOQA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.586902271 (81h46m58.284470587s) [noise=true , address=185.167.204.205:4011 , piece_id=6QG7AY7VU2OZQQ2BWEN4JXGPSV6OJUIGMA7BPFS2MRGXAF2YEZ3Q, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.587190569 (81h46m58.284758886s) [noise=true , address=81.171.14.70:4030 , piece_id=SKV6A7MNHALYNDYB32PFKZZY2KFX2H6EB5ABXUMBZMCX5UV7M3EA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:11.070889191 (81h46m58.768457497s) [upload=71769]: Next returned nil segment Dec 9 01:16:11.070900042 (81h46m58.768468347s) [upload=71769]: Waiting for error group managing segments... Dec 9 01:16:11.306441373 (81h46m59.004009689s) [noise=true , address=185.167.204.205:4006 , piece_id=VUWLGRP6NRCJZAFQ3DBXFEGX6USEY2WRVJOBXWDX2CLJG3Z35K7Q, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.677434182 (81h46m59.375002498s) [noise=true , address=81.171.14.70:4029 , piece_id=6DAIEFNGDOHF7WUK7GOVTLZN6O3KH3S3QB2AKYCQSPJ3ML6EYE5A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.766219989 (81h46m59.463788305s) [noise=true , address=89.149.201.147:4004 , piece_id=H2NLAGUIGKVFE5FB7EZRAEN7JKWEYENEEE52NB4T655UIJMBHJSA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.779958836 (81h46m59.477527152s) [noise=true , address=81.171.14.70:4030 , piece_id=SKV6A7MNHALYNDYB32PFKZZY2KFX2H6EB5ABXUMBZMCX5UV7M3EA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.795060940 (81h46m59.492629245s) [noise=true , address=81.171.14.70:4038 , piece_id=2K5WRMG5L3F4G24IO3UH4EKSFS62UVLWX64GMLQCZY266CEG4IYA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.799263681 (81h46m59.496831997s) [noise=true , address=81.171.14.70:4044 , piece_id=FAWA7ZQLX3QCHTURK3WMCETCS5LUVYFOSTF7JKMHP6CXTALDSBXA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.826689817 (81h46m59.524258123s) [noise=true , address=89.149.201.147:4019 , piece_id=IFQZWRCXGQJE7KK7WFHUWPLYSERMQHYNXGATKRSLK7FPSBRFYLRQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.994726298 (81h46m59.692294605s) [noise=true , address=185.167.204.206:4045 , piece_id=EZNMLAFRR5D3P6AHPGJEG62ZOVJCGS7AE2G23EXF7J2ZOXJWFJ7A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.016556801 (81h46m59.714125107s) [noise=true , address=185.167.204.206:4043 , piece_id=A35E7GZE4ATEJB2BCNV76WFSZV6XFCPVLD2PM3Z5H37NYUM5FDXQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.088723221 (81h46m59.786291558s) [noise=true , address=81.171.14.70:4041 , piece_id=6SEG5ZF43PAE367OEPMR62E35ARUTZIKT6WBRYAJ3GQKQRDXOI7A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.108763619 (81h46m59.806331956s) [noise=true , address=185.167.204.206:4039 , piece_id=6EHHODKFZJAR4UVH63LDKL6VFXTXUUYMDBI4JHNOVPPETNIPE35A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.110609809 (81h46m59.808178144s) [noise=true , address=81.171.14.70:4046 , piece_id=HRZLGUFGQQ4LWBFQKOR2HBTLNIQSWRGYZGJWHFVTT7KWJKXTRPTQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.136656809 (81h46m59.834225146s) [noise=true , address=185.167.204.206:4057 , piece_id=XBVYRI2MF3LNAMFSOOQQLZGOXDAAIM6KSQO6DIPVDN3IHEW5JVMQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.190317096 (81h46m59.887885432s) [noise=true , address=81.171.14.70:4034 , piece_id=GHSKMR2MYLHMKRQBNZMWHTOICWQIRRJLYVQQISQOGMT45PF7SSSQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.192207097 (81h46m59.889775413s) [noise=true , address=81.171.14.70:4048 , piece_id=BFEGOKTJDFRQKCTBIXE6UD6EIX7N6QV4PICUM3MGO4EIUMRKFITA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.204614155 (81h46m59.902182492s) [noise=true , address=185.167.204.205:4036 , piece_id=2N4ZAUATTVLIRJAXGL34KBKQTN4VDEBAPTX7HK4G3IQLNU7KEOQA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.205185243 (81h46m59.902753578s) [noise=true , address=185.167.204.205:4011 , piece_id=6QG7AY7VU2OZQQ2BWEN4JXGPSV6OJUIGMA7BPFS2MRGXAF2YEZ3Q, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.231785917 (81h46m59.929354223s) [noise=true , address=89.149.201.147:4016 , piece_id=CI3OOTGSMU2BU5EVPBPQRVAI2B6OTPY47SVW4FLV5LFT37ZAIQLA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.266589316 (81h46m59.964157612s) [noise=true , address=89.149.201.147:4022 , piece_id=Z6OHJP6I3BYDMEPDHFOPJQQN2PPWAR6IXGCVQROKJVSHUHUTE5IA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.276030935 (81h46m59.973599241s) [noise=true , address=89.149.201.147:4024 , piece_id=LBJZHAJWAMUEMMLT25TXSPAGCRKDEWDXUKPLIEYKMXCWOP2NFHCQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.315883226 (81h47m0.013451532s) [noise=true , address=185.167.204.206:4071 , piece_id=KA4HORF25UFNNH4GWCQQUSWVJCSHDBLVR3K6I77AQ7UW2AUSSBRQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.487801117 (81h47m0.185369424s) [noise=true , address=81.171.14.70:4025 , piece_id=C7OUWVW2X67WYCR7HWPAFOQO7EHTMURUNYS4JC65C3LK4YSD4MAQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.581328313 (81h47m0.278896619s) [noise=true , address=81.171.14.70:4037 , piece_id=XXALUHDRGQNICTX2JFRO7WMUFTGJBOPKPRLOXGWWBQSVAW4OM4PQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.586190246 (81h47m0.283758553s) [noise=true , address=89.149.201.147:4003 , piece_id=ZKRRIMCDW7JDADUMF3XOQPQW7AYVJ66ZQOVPLH5Y7E2ASHHFISYA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.943766260 (81h47m0.641334577s) [noise=true , address=89.149.201.147:4008 , piece_id=HVYVZARPZ5WM3ISOGULFNA6WPCKKC3PRIMCE7226JTYCPDVXMUBA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:13.223084536 (81h47m0.920652852s) [noise=true , address=185.167.204.205:4001 , piece_id=VY7CIY72VGBGZSFEI2H43N4ELABFIXCICEVQNEZB5TWRM2WMZSEQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:13.223109182 (81h47m0.920677487s) [seg_pos={0 0}, upload=71769]: Segment reached optimal threshold of 26 pieces. Dec 9 01:16:13.223242000 (81h47m0.920810306s) [noise=true , address=81.171.14.70:4045 , piece_id=FD2J6MOI2FZWTDD4JCGAEQHGK6AUQNEQWLQ7PRGCJIV7L4VEXWDQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCMeEEkwoqbPeyENfJt6nahQz2h3yW7st1TtvoY6KDKZqUGmS): protocol: expected piece hash; context canceled; EOF Dec 9 01:16:13.223247510 (81h47m0.920815826s) [noise=true , address=81.171.14.70:4045 , piece_id=FD2J6MOI2FZWTDD4JCGAEQHGK6AUQNEQWLQ7PRGCJIV7L4VEXWDQ, seg_pos={0 0}, upload=71769]: Long tail cancellation due to optimal threshold Dec 9 01:16:13.223590229 (81h47m0.921158566s) [noise=true , address=81.171.14.70:4033 , piece_id=ETKLQ4F4LJ2Y75WZS4V6D3BP3SGX43ENIJZES7EVORZFHGBBLNWA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 9 01:16:13.223600700 (81h47m0.921169005s) [noise=true , address=81.171.14.70:4033 , piece_id=ETKLQ4F4LJ2Y75WZS4V6D3BP3SGX43ENIJZES7EVORZFHGBBLNWA, seg_pos={0 0}, upload=71769]: Long tail cancellation due to optimal threshold Dec 9 01:16:13.223618643 (81h47m0.921186959s) [seg_pos={0 0}, upload=71769]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:16:13.223676281 (81h47m0.921244597s) [upload=71769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:16:13.223680419 (81h47m0.921248725s) [upload=71769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:16:13.372835035 (81h47m1.070403351s) [upload=71769]: Done uploading segments... Dec 9 01:16:13.372850073 (81h47m1.070418389s) [upload=71769]: Upload finished. err: <nil> Duration: 7.996154664s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 435.043778ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:18:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 435.06607ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 412.791435ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:18:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 412.807694ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 15 out of 60 is failed (threshold=3) [....X...XX.....XX..X...X.....X.X.......X...X.....X.X.X...X..] duration: 11.299248267s trace: https://jaeger.dev.storj.io/trace/14a5491891bee05e FAIL (2025-12-09 01:17:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== address=185.167.204.205:4001 , piece_id=VY7CIY72VGBGZSFEI2H43N4ELABFIXCICEVQNEZB5TWRM2WMZSEQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.584421346 (81h46m58.281989662s) [noise=true , address=89.149.201.147:4008 , piece_id=HVYVZARPZ5WM3ISOGULFNA6WPCKKC3PRIMCE7226JTYCPDVXMUBA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.584711679 (81h46m58.282279995s) [noise=true , address=81.171.14.70:4034 , piece_id=GHSKMR2MYLHMKRQBNZMWHTOICWQIRRJLYVQQISQOGMT45PF7SSSQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585013122 (81h46m58.282581427s) [noise=true , address=81.171.14.70:4045 , piece_id=FD2J6MOI2FZWTDD4JCGAEQHGK6AUQNEQWLQ7PRGCJIV7L4VEXWDQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585303003 (81h46m58.282871309s) [noise=true , address=81.171.14.70:4025 , piece_id=C7OUWVW2X67WYCR7HWPAFOQO7EHTMURUNYS4JC65C3LK4YSD4MAQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585619755 (81h46m58.283188071s) [noise=true , address=185.167.204.206:4071 , piece_id=KA4HORF25UFNNH4GWCQQUSWVJCSHDBLVR3K6I77AQ7UW2AUSSBRQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585940374 (81h46m58.283508679s) [noise=true , address=185.167.204.206:4043 , piece_id=A35E7GZE4ATEJB2BCNV76WFSZV6XFCPVLD2PM3Z5H37NYUM5FDXQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.586287633 (81h46m58.283855949s) [noise=true , address=89.149.201.147:4016 , piece_id=CI3OOTGSMU2BU5EVPBPQRVAI2B6OTPY47SVW4FLV5LFT37ZAIQLA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.586579107 (81h46m58.284147423s) [noise=true , address=185.167.204.205:4036 , piece_id=2N4ZAUATTVLIRJAXGL34KBKQTN4VDEBAPTX7HK4G3IQLNU7KEOQA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.586902271 (81h46m58.284470587s) [noise=true , address=185.167.204.205:4011 , piece_id=6QG7AY7VU2OZQQ2BWEN4JXGPSV6OJUIGMA7BPFS2MRGXAF2YEZ3Q, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.587190569 (81h46m58.284758886s) [noise=true , address=81.171.14.70:4030 , piece_id=SKV6A7MNHALYNDYB32PFKZZY2KFX2H6EB5ABXUMBZMCX5UV7M3EA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:11.070889191 (81h46m58.768457497s) [upload=71769]: Next returned nil segment Dec 9 01:16:11.070900042 (81h46m58.768468347s) [upload=71769]: Waiting for error group managing segments... Dec 9 01:16:11.306441373 (81h46m59.004009689s) [noise=true , address=185.167.204.205:4006 , piece_id=VUWLGRP6NRCJZAFQ3DBXFEGX6USEY2WRVJOBXWDX2CLJG3Z35K7Q, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.677434182 (81h46m59.375002498s) [noise=true , address=81.171.14.70:4029 , piece_id=6DAIEFNGDOHF7WUK7GOVTLZN6O3KH3S3QB2AKYCQSPJ3ML6EYE5A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.766219989 (81h46m59.463788305s) [noise=true , address=89.149.201.147:4004 , piece_id=H2NLAGUIGKVFE5FB7EZRAEN7JKWEYENEEE52NB4T655UIJMBHJSA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.779958836 (81h46m59.477527152s) [noise=true , address=81.171.14.70:4030 , piece_id=SKV6A7MNHALYNDYB32PFKZZY2KFX2H6EB5ABXUMBZMCX5UV7M3EA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.795060940 (81h46m59.492629245s) [noise=true , address=81.171.14.70:4038 , piece_id=2K5WRMG5L3F4G24IO3UH4EKSFS62UVLWX64GMLQCZY266CEG4IYA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.799263681 (81h46m59.496831997s) [noise=true , address=81.171.14.70:4044 , piece_id=FAWA7ZQLX3QCHTURK3WMCETCS5LUVYFOSTF7JKMHP6CXTALDSBXA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.826689817 (81h46m59.524258123s) [noise=true , address=89.149.201.147:4019 , piece_id=IFQZWRCXGQJE7KK7WFHUWPLYSERMQHYNXGATKRSLK7FPSBRFYLRQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.994726298 (81h46m59.692294605s) [noise=true , address=185.167.204.206:4045 , piece_id=EZNMLAFRR5D3P6AHPGJEG62ZOVJCGS7AE2G23EXF7J2ZOXJWFJ7A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.016556801 (81h46m59.714125107s) [noise=true , address=185.167.204.206:4043 , piece_id=A35E7GZE4ATEJB2BCNV76WFSZV6XFCPVLD2PM3Z5H37NYUM5FDXQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.088723221 (81h46m59.786291558s) [noise=true , address=81.171.14.70:4041 , piece_id=6SEG5ZF43PAE367OEPMR62E35ARUTZIKT6WBRYAJ3GQKQRDXOI7A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.108763619 (81h46m59.806331956s) [noise=true , address=185.167.204.206:4039 , piece_id=6EHHODKFZJAR4UVH63LDKL6VFXTXUUYMDBI4JHNOVPPETNIPE35A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.110609809 (81h46m59.808178144s) [noise=true , address=81.171.14.70:4046 , piece_id=HRZLGUFGQQ4LWBFQKOR2HBTLNIQSWRGYZGJWHFVTT7KWJKXTRPTQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.136656809 (81h46m59.834225146s) [noise=true , address=185.167.204.206:4057 , piece_id=XBVYRI2MF3LNAMFSOOQQLZGOXDAAIM6KSQO6DIPVDN3IHEW5JVMQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.190317096 (81h46m59.887885432s) [noise=true , address=81.171.14.70:4034 , piece_id=GHSKMR2MYLHMKRQBNZMWHTOICWQIRRJLYVQQISQOGMT45PF7SSSQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.192207097 (81h46m59.889775413s) [noise=true , address=81.171.14.70:4048 , piece_id=BFEGOKTJDFRQKCTBIXE6UD6EIX7N6QV4PICUM3MGO4EIUMRKFITA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.204614155 (81h46m59.902182492s) [noise=true , address=185.167.204.205:4036 , piece_id=2N4ZAUATTVLIRJAXGL34KBKQTN4VDEBAPTX7HK4G3IQLNU7KEOQA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.205185243 (81h46m59.902753578s) [noise=true , address=185.167.204.205:4011 , piece_id=6QG7AY7VU2OZQQ2BWEN4JXGPSV6OJUIGMA7BPFS2MRGXAF2YEZ3Q, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.231785917 (81h46m59.929354223s) [noise=true , address=89.149.201.147:4016 , piece_id=CI3OOTGSMU2BU5EVPBPQRVAI2B6OTPY47SVW4FLV5LFT37ZAIQLA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.266589316 (81h46m59.964157612s) [noise=true , address=89.149.201.147:4022 , piece_id=Z6OHJP6I3BYDMEPDHFOPJQQN2PPWAR6IXGCVQROKJVSHUHUTE5IA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.276030935 (81h46m59.973599241s) [noise=true , address=89.149.201.147:4024 , piece_id=LBJZHAJWAMUEMMLT25TXSPAGCRKDEWDXUKPLIEYKMXCWOP2NFHCQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.315883226 (81h47m0.013451532s) [noise=true , address=185.167.204.206:4071 , piece_id=KA4HORF25UFNNH4GWCQQUSWVJCSHDBLVR3K6I77AQ7UW2AUSSBRQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.487801117 (81h47m0.185369424s) [noise=true , address=81.171.14.70:4025 , piece_id=C7OUWVW2X67WYCR7HWPAFOQO7EHTMURUNYS4JC65C3LK4YSD4MAQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.581328313 (81h47m0.278896619s) [noise=true , address=81.171.14.70:4037 , piece_id=XXALUHDRGQNICTX2JFRO7WMUFTGJBOPKPRLOXGWWBQSVAW4OM4PQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.586190246 (81h47m0.283758553s) [noise=true , address=89.149.201.147:4003 , piece_id=ZKRRIMCDW7JDADUMF3XOQPQW7AYVJ66ZQOVPLH5Y7E2ASHHFISYA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.943766260 (81h47m0.641334577s) [noise=true , address=89.149.201.147:4008 , piece_id=HVYVZARPZ5WM3ISOGULFNA6WPCKKC3PRIMCE7226JTYCPDVXMUBA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:13.223084536 (81h47m0.920652852s) [noise=true , address=185.167.204.205:4001 , piece_id=VY7CIY72VGBGZSFEI2H43N4ELABFIXCICEVQNEZB5TWRM2WMZSEQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:13.223109182 (81h47m0.920677487s) [seg_pos={0 0}, upload=71769]: Segment reached optimal threshold of 26 pieces. Dec 9 01:16:13.223242000 (81h47m0.920810306s) [noise=true , address=81.171.14.70:4045 , piece_id=FD2J6MOI2FZWTDD4JCGAEQHGK6AUQNEQWLQ7PRGCJIV7L4VEXWDQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCMeEEkwoqbPeyENfJt6nahQz2h3yW7st1TtvoY6KDKZqUGmS): protocol: expected piece hash; context canceled; EOF Dec 9 01:16:13.223247510 (81h47m0.920815826s) [noise=true , address=81.171.14.70:4045 , piece_id=FD2J6MOI2FZWTDD4JCGAEQHGK6AUQNEQWLQ7PRGCJIV7L4VEXWDQ, seg_pos={0 0}, upload=71769]: Long tail cancellation due to optimal threshold Dec 9 01:16:13.223590229 (81h47m0.921158566s) [noise=true , address=81.171.14.70:4033 , piece_id=ETKLQ4F4LJ2Y75WZS4V6D3BP3SGX43ENIJZES7EVORZFHGBBLNWA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 9 01:16:13.223600700 (81h47m0.921169005s) [noise=true , address=81.171.14.70:4033 , piece_id=ETKLQ4F4LJ2Y75WZS4V6D3BP3SGX43ENIJZES7EVORZFHGBBLNWA, seg_pos={0 0}, upload=71769]: Long tail cancellation due to optimal threshold Dec 9 01:16:13.223618643 (81h47m0.921186959s) [seg_pos={0 0}, upload=71769]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:16:13.223676281 (81h47m0.921244597s) [upload=71769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:16:13.223680419 (81h47m0.921248725s) [upload=71769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:16:13.372835035 (81h47m1.070403351s) [upload=71769]: Done uploading segments... Dec 9 01:16:13.372850073 (81h47m1.070418389s) [upload=71769]: Upload finished. err: <nil> Duration: 11.299264628s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 15 out of 60 is failed (threshold=3) [.....X...XX.....XX..X...X.....X.X.......X...X.....X.X.X...X.] duration: 30.380128657s trace: https://jaeger.dev.storj.io/trace/14a5491891bee05e FAIL (2025-12-09 01:17:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== address=185.167.204.205:4001 , piece_id=VY7CIY72VGBGZSFEI2H43N4ELABFIXCICEVQNEZB5TWRM2WMZSEQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.584421346 (81h46m58.281989662s) [noise=true , address=89.149.201.147:4008 , piece_id=HVYVZARPZ5WM3ISOGULFNA6WPCKKC3PRIMCE7226JTYCPDVXMUBA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.584711679 (81h46m58.282279995s) [noise=true , address=81.171.14.70:4034 , piece_id=GHSKMR2MYLHMKRQBNZMWHTOICWQIRRJLYVQQISQOGMT45PF7SSSQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585013122 (81h46m58.282581427s) [noise=true , address=81.171.14.70:4045 , piece_id=FD2J6MOI2FZWTDD4JCGAEQHGK6AUQNEQWLQ7PRGCJIV7L4VEXWDQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585303003 (81h46m58.282871309s) [noise=true , address=81.171.14.70:4025 , piece_id=C7OUWVW2X67WYCR7HWPAFOQO7EHTMURUNYS4JC65C3LK4YSD4MAQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585619755 (81h46m58.283188071s) [noise=true , address=185.167.204.206:4071 , piece_id=KA4HORF25UFNNH4GWCQQUSWVJCSHDBLVR3K6I77AQ7UW2AUSSBRQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585940374 (81h46m58.283508679s) [noise=true , address=185.167.204.206:4043 , piece_id=A35E7GZE4ATEJB2BCNV76WFSZV6XFCPVLD2PM3Z5H37NYUM5FDXQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.586287633 (81h46m58.283855949s) [noise=true , address=89.149.201.147:4016 , piece_id=CI3OOTGSMU2BU5EVPBPQRVAI2B6OTPY47SVW4FLV5LFT37ZAIQLA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.586579107 (81h46m58.284147423s) [noise=true , address=185.167.204.205:4036 , piece_id=2N4ZAUATTVLIRJAXGL34KBKQTN4VDEBAPTX7HK4G3IQLNU7KEOQA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.586902271 (81h46m58.284470587s) [noise=true , address=185.167.204.205:4011 , piece_id=6QG7AY7VU2OZQQ2BWEN4JXGPSV6OJUIGMA7BPFS2MRGXAF2YEZ3Q, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.587190569 (81h46m58.284758886s) [noise=true , address=81.171.14.70:4030 , piece_id=SKV6A7MNHALYNDYB32PFKZZY2KFX2H6EB5ABXUMBZMCX5UV7M3EA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:11.070889191 (81h46m58.768457497s) [upload=71769]: Next returned nil segment Dec 9 01:16:11.070900042 (81h46m58.768468347s) [upload=71769]: Waiting for error group managing segments... Dec 9 01:16:11.306441373 (81h46m59.004009689s) [noise=true , address=185.167.204.205:4006 , piece_id=VUWLGRP6NRCJZAFQ3DBXFEGX6USEY2WRVJOBXWDX2CLJG3Z35K7Q, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.677434182 (81h46m59.375002498s) [noise=true , address=81.171.14.70:4029 , piece_id=6DAIEFNGDOHF7WUK7GOVTLZN6O3KH3S3QB2AKYCQSPJ3ML6EYE5A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.766219989 (81h46m59.463788305s) [noise=true , address=89.149.201.147:4004 , piece_id=H2NLAGUIGKVFE5FB7EZRAEN7JKWEYENEEE52NB4T655UIJMBHJSA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.779958836 (81h46m59.477527152s) [noise=true , address=81.171.14.70:4030 , piece_id=SKV6A7MNHALYNDYB32PFKZZY2KFX2H6EB5ABXUMBZMCX5UV7M3EA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.795060940 (81h46m59.492629245s) [noise=true , address=81.171.14.70:4038 , piece_id=2K5WRMG5L3F4G24IO3UH4EKSFS62UVLWX64GMLQCZY266CEG4IYA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.799263681 (81h46m59.496831997s) [noise=true , address=81.171.14.70:4044 , piece_id=FAWA7ZQLX3QCHTURK3WMCETCS5LUVYFOSTF7JKMHP6CXTALDSBXA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.826689817 (81h46m59.524258123s) [noise=true , address=89.149.201.147:4019 , piece_id=IFQZWRCXGQJE7KK7WFHUWPLYSERMQHYNXGATKRSLK7FPSBRFYLRQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.994726298 (81h46m59.692294605s) [noise=true , address=185.167.204.206:4045 , piece_id=EZNMLAFRR5D3P6AHPGJEG62ZOVJCGS7AE2G23EXF7J2ZOXJWFJ7A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.016556801 (81h46m59.714125107s) [noise=true , address=185.167.204.206:4043 , piece_id=A35E7GZE4ATEJB2BCNV76WFSZV6XFCPVLD2PM3Z5H37NYUM5FDXQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.088723221 (81h46m59.786291558s) [noise=true , address=81.171.14.70:4041 , piece_id=6SEG5ZF43PAE367OEPMR62E35ARUTZIKT6WBRYAJ3GQKQRDXOI7A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.108763619 (81h46m59.806331956s) [noise=true , address=185.167.204.206:4039 , piece_id=6EHHODKFZJAR4UVH63LDKL6VFXTXUUYMDBI4JHNOVPPETNIPE35A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.110609809 (81h46m59.808178144s) [noise=true , address=81.171.14.70:4046 , piece_id=HRZLGUFGQQ4LWBFQKOR2HBTLNIQSWRGYZGJWHFVTT7KWJKXTRPTQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.136656809 (81h46m59.834225146s) [noise=true , address=185.167.204.206:4057 , piece_id=XBVYRI2MF3LNAMFSOOQQLZGOXDAAIM6KSQO6DIPVDN3IHEW5JVMQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.190317096 (81h46m59.887885432s) [noise=true , address=81.171.14.70:4034 , piece_id=GHSKMR2MYLHMKRQBNZMWHTOICWQIRRJLYVQQISQOGMT45PF7SSSQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.192207097 (81h46m59.889775413s) [noise=true , address=81.171.14.70:4048 , piece_id=BFEGOKTJDFRQKCTBIXE6UD6EIX7N6QV4PICUM3MGO4EIUMRKFITA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.204614155 (81h46m59.902182492s) [noise=true , address=185.167.204.205:4036 , piece_id=2N4ZAUATTVLIRJAXGL34KBKQTN4VDEBAPTX7HK4G3IQLNU7KEOQA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.205185243 (81h46m59.902753578s) [noise=true , address=185.167.204.205:4011 , piece_id=6QG7AY7VU2OZQQ2BWEN4JXGPSV6OJUIGMA7BPFS2MRGXAF2YEZ3Q, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.231785917 (81h46m59.929354223s) [noise=true , address=89.149.201.147:4016 , piece_id=CI3OOTGSMU2BU5EVPBPQRVAI2B6OTPY47SVW4FLV5LFT37ZAIQLA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.266589316 (81h46m59.964157612s) [noise=true , address=89.149.201.147:4022 , piece_id=Z6OHJP6I3BYDMEPDHFOPJQQN2PPWAR6IXGCVQROKJVSHUHUTE5IA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.276030935 (81h46m59.973599241s) [noise=true , address=89.149.201.147:4024 , piece_id=LBJZHAJWAMUEMMLT25TXSPAGCRKDEWDXUKPLIEYKMXCWOP2NFHCQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.315883226 (81h47m0.013451532s) [noise=true , address=185.167.204.206:4071 , piece_id=KA4HORF25UFNNH4GWCQQUSWVJCSHDBLVR3K6I77AQ7UW2AUSSBRQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.487801117 (81h47m0.185369424s) [noise=true , address=81.171.14.70:4025 , piece_id=C7OUWVW2X67WYCR7HWPAFOQO7EHTMURUNYS4JC65C3LK4YSD4MAQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.581328313 (81h47m0.278896619s) [noise=true , address=81.171.14.70:4037 , piece_id=XXALUHDRGQNICTX2JFRO7WMUFTGJBOPKPRLOXGWWBQSVAW4OM4PQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.586190246 (81h47m0.283758553s) [noise=true , address=89.149.201.147:4003 , piece_id=ZKRRIMCDW7JDADUMF3XOQPQW7AYVJ66ZQOVPLH5Y7E2ASHHFISYA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.943766260 (81h47m0.641334577s) [noise=true , address=89.149.201.147:4008 , piece_id=HVYVZARPZ5WM3ISOGULFNA6WPCKKC3PRIMCE7226JTYCPDVXMUBA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:13.223084536 (81h47m0.920652852s) [noise=true , address=185.167.204.205:4001 , piece_id=VY7CIY72VGBGZSFEI2H43N4ELABFIXCICEVQNEZB5TWRM2WMZSEQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:13.223109182 (81h47m0.920677487s) [seg_pos={0 0}, upload=71769]: Segment reached optimal threshold of 26 pieces. Dec 9 01:16:13.223242000 (81h47m0.920810306s) [noise=true , address=81.171.14.70:4045 , piece_id=FD2J6MOI2FZWTDD4JCGAEQHGK6AUQNEQWLQ7PRGCJIV7L4VEXWDQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCMeEEkwoqbPeyENfJt6nahQz2h3yW7st1TtvoY6KDKZqUGmS): protocol: expected piece hash; context canceled; EOF Dec 9 01:16:13.223247510 (81h47m0.920815826s) [noise=true , address=81.171.14.70:4045 , piece_id=FD2J6MOI2FZWTDD4JCGAEQHGK6AUQNEQWLQ7PRGCJIV7L4VEXWDQ, seg_pos={0 0}, upload=71769]: Long tail cancellation due to optimal threshold Dec 9 01:16:13.223590229 (81h47m0.921158566s) [noise=true , address=81.171.14.70:4033 , piece_id=ETKLQ4F4LJ2Y75WZS4V6D3BP3SGX43ENIJZES7EVORZFHGBBLNWA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 9 01:16:13.223600700 (81h47m0.921169005s) [noise=true , address=81.171.14.70:4033 , piece_id=ETKLQ4F4LJ2Y75WZS4V6D3BP3SGX43ENIJZES7EVORZFHGBBLNWA, seg_pos={0 0}, upload=71769]: Long tail cancellation due to optimal threshold Dec 9 01:16:13.223618643 (81h47m0.921186959s) [seg_pos={0 0}, upload=71769]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:16:13.223676281 (81h47m0.921244597s) [upload=71769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:16:13.223680419 (81h47m0.921248725s) [upload=71769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:16:13.372835035 (81h47m1.070403351s) [upload=71769]: Done uploading segments... Dec 9 01:16:13.372850073 (81h47m1.070418389s) [upload=71769]: Upload finished. err: <nil> Duration: 30.380146991s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 429.903585ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:17:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 429.919565ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 433.057776ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:17:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 433.085458ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 15 out of 60 is failed (threshold=3) [......X...XX.....XX..X...X.....X.X.......X...X.....X.X.X...X] duration: 35.6094448s trace: https://jaeger.dev.storj.io/trace/14a5491891bee05e FAIL (2025-12-09 01:16:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== address=185.167.204.205:4001 , piece_id=VY7CIY72VGBGZSFEI2H43N4ELABFIXCICEVQNEZB5TWRM2WMZSEQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.584421346 (81h46m58.281989662s) [noise=true , address=89.149.201.147:4008 , piece_id=HVYVZARPZ5WM3ISOGULFNA6WPCKKC3PRIMCE7226JTYCPDVXMUBA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.584711679 (81h46m58.282279995s) [noise=true , address=81.171.14.70:4034 , piece_id=GHSKMR2MYLHMKRQBNZMWHTOICWQIRRJLYVQQISQOGMT45PF7SSSQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585013122 (81h46m58.282581427s) [noise=true , address=81.171.14.70:4045 , piece_id=FD2J6MOI2FZWTDD4JCGAEQHGK6AUQNEQWLQ7PRGCJIV7L4VEXWDQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585303003 (81h46m58.282871309s) [noise=true , address=81.171.14.70:4025 , piece_id=C7OUWVW2X67WYCR7HWPAFOQO7EHTMURUNYS4JC65C3LK4YSD4MAQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585619755 (81h46m58.283188071s) [noise=true , address=185.167.204.206:4071 , piece_id=KA4HORF25UFNNH4GWCQQUSWVJCSHDBLVR3K6I77AQ7UW2AUSSBRQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.585940374 (81h46m58.283508679s) [noise=true , address=185.167.204.206:4043 , piece_id=A35E7GZE4ATEJB2BCNV76WFSZV6XFCPVLD2PM3Z5H37NYUM5FDXQ, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.586287633 (81h46m58.283855949s) [noise=true , address=89.149.201.147:4016 , piece_id=CI3OOTGSMU2BU5EVPBPQRVAI2B6OTPY47SVW4FLV5LFT37ZAIQLA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.586579107 (81h46m58.284147423s) [noise=true , address=185.167.204.205:4036 , piece_id=2N4ZAUATTVLIRJAXGL34KBKQTN4VDEBAPTX7HK4G3IQLNU7KEOQA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.586902271 (81h46m58.284470587s) [noise=true , address=185.167.204.205:4011 , piece_id=6QG7AY7VU2OZQQ2BWEN4JXGPSV6OJUIGMA7BPFS2MRGXAF2YEZ3Q, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:10.587190569 (81h46m58.284758886s) [noise=true , address=81.171.14.70:4030 , piece_id=SKV6A7MNHALYNDYB32PFKZZY2KFX2H6EB5ABXUMBZMCX5UV7M3EA, seg_pos={0 0}, upload=71769]: Uploading piece... Dec 9 01:16:11.070889191 (81h46m58.768457497s) [upload=71769]: Next returned nil segment Dec 9 01:16:11.070900042 (81h46m58.768468347s) [upload=71769]: Waiting for error group managing segments... Dec 9 01:16:11.306441373 (81h46m59.004009689s) [noise=true , address=185.167.204.205:4006 , piece_id=VUWLGRP6NRCJZAFQ3DBXFEGX6USEY2WRVJOBXWDX2CLJG3Z35K7Q, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.677434182 (81h46m59.375002498s) [noise=true , address=81.171.14.70:4029 , piece_id=6DAIEFNGDOHF7WUK7GOVTLZN6O3KH3S3QB2AKYCQSPJ3ML6EYE5A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.766219989 (81h46m59.463788305s) [noise=true , address=89.149.201.147:4004 , piece_id=H2NLAGUIGKVFE5FB7EZRAEN7JKWEYENEEE52NB4T655UIJMBHJSA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.779958836 (81h46m59.477527152s) [noise=true , address=81.171.14.70:4030 , piece_id=SKV6A7MNHALYNDYB32PFKZZY2KFX2H6EB5ABXUMBZMCX5UV7M3EA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.795060940 (81h46m59.492629245s) [noise=true , address=81.171.14.70:4038 , piece_id=2K5WRMG5L3F4G24IO3UH4EKSFS62UVLWX64GMLQCZY266CEG4IYA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.799263681 (81h46m59.496831997s) [noise=true , address=81.171.14.70:4044 , piece_id=FAWA7ZQLX3QCHTURK3WMCETCS5LUVYFOSTF7JKMHP6CXTALDSBXA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.826689817 (81h46m59.524258123s) [noise=true , address=89.149.201.147:4019 , piece_id=IFQZWRCXGQJE7KK7WFHUWPLYSERMQHYNXGATKRSLK7FPSBRFYLRQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:11.994726298 (81h46m59.692294605s) [noise=true , address=185.167.204.206:4045 , piece_id=EZNMLAFRR5D3P6AHPGJEG62ZOVJCGS7AE2G23EXF7J2ZOXJWFJ7A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.016556801 (81h46m59.714125107s) [noise=true , address=185.167.204.206:4043 , piece_id=A35E7GZE4ATEJB2BCNV76WFSZV6XFCPVLD2PM3Z5H37NYUM5FDXQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.088723221 (81h46m59.786291558s) [noise=true , address=81.171.14.70:4041 , piece_id=6SEG5ZF43PAE367OEPMR62E35ARUTZIKT6WBRYAJ3GQKQRDXOI7A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.108763619 (81h46m59.806331956s) [noise=true , address=185.167.204.206:4039 , piece_id=6EHHODKFZJAR4UVH63LDKL6VFXTXUUYMDBI4JHNOVPPETNIPE35A, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.110609809 (81h46m59.808178144s) [noise=true , address=81.171.14.70:4046 , piece_id=HRZLGUFGQQ4LWBFQKOR2HBTLNIQSWRGYZGJWHFVTT7KWJKXTRPTQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.136656809 (81h46m59.834225146s) [noise=true , address=185.167.204.206:4057 , piece_id=XBVYRI2MF3LNAMFSOOQQLZGOXDAAIM6KSQO6DIPVDN3IHEW5JVMQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.190317096 (81h46m59.887885432s) [noise=true , address=81.171.14.70:4034 , piece_id=GHSKMR2MYLHMKRQBNZMWHTOICWQIRRJLYVQQISQOGMT45PF7SSSQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.192207097 (81h46m59.889775413s) [noise=true , address=81.171.14.70:4048 , piece_id=BFEGOKTJDFRQKCTBIXE6UD6EIX7N6QV4PICUM3MGO4EIUMRKFITA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.204614155 (81h46m59.902182492s) [noise=true , address=185.167.204.205:4036 , piece_id=2N4ZAUATTVLIRJAXGL34KBKQTN4VDEBAPTX7HK4G3IQLNU7KEOQA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.205185243 (81h46m59.902753578s) [noise=true , address=185.167.204.205:4011 , piece_id=6QG7AY7VU2OZQQ2BWEN4JXGPSV6OJUIGMA7BPFS2MRGXAF2YEZ3Q, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.231785917 (81h46m59.929354223s) [noise=true , address=89.149.201.147:4016 , piece_id=CI3OOTGSMU2BU5EVPBPQRVAI2B6OTPY47SVW4FLV5LFT37ZAIQLA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.266589316 (81h46m59.964157612s) [noise=true , address=89.149.201.147:4022 , piece_id=Z6OHJP6I3BYDMEPDHFOPJQQN2PPWAR6IXGCVQROKJVSHUHUTE5IA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.276030935 (81h46m59.973599241s) [noise=true , address=89.149.201.147:4024 , piece_id=LBJZHAJWAMUEMMLT25TXSPAGCRKDEWDXUKPLIEYKMXCWOP2NFHCQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.315883226 (81h47m0.013451532s) [noise=true , address=185.167.204.206:4071 , piece_id=KA4HORF25UFNNH4GWCQQUSWVJCSHDBLVR3K6I77AQ7UW2AUSSBRQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.487801117 (81h47m0.185369424s) [noise=true , address=81.171.14.70:4025 , piece_id=C7OUWVW2X67WYCR7HWPAFOQO7EHTMURUNYS4JC65C3LK4YSD4MAQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.581328313 (81h47m0.278896619s) [noise=true , address=81.171.14.70:4037 , piece_id=XXALUHDRGQNICTX2JFRO7WMUFTGJBOPKPRLOXGWWBQSVAW4OM4PQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.586190246 (81h47m0.283758553s) [noise=true , address=89.149.201.147:4003 , piece_id=ZKRRIMCDW7JDADUMF3XOQPQW7AYVJ66ZQOVPLH5Y7E2ASHHFISYA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:12.943766260 (81h47m0.641334577s) [noise=true , address=89.149.201.147:4008 , piece_id=HVYVZARPZ5WM3ISOGULFNA6WPCKKC3PRIMCE7226JTYCPDVXMUBA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:13.223084536 (81h47m0.920652852s) [noise=true , address=185.167.204.205:4001 , piece_id=VY7CIY72VGBGZSFEI2H43N4ELABFIXCICEVQNEZB5TWRM2WMZSEQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: <nil> Dec 9 01:16:13.223109182 (81h47m0.920677487s) [seg_pos={0 0}, upload=71769]: Segment reached optimal threshold of 26 pieces. Dec 9 01:16:13.223242000 (81h47m0.920810306s) [noise=true , address=81.171.14.70:4045 , piece_id=FD2J6MOI2FZWTDD4JCGAEQHGK6AUQNEQWLQ7PRGCJIV7L4VEXWDQ, seg_pos={0 0}, upload=71769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCMeEEkwoqbPeyENfJt6nahQz2h3yW7st1TtvoY6KDKZqUGmS): protocol: expected piece hash; context canceled; EOF Dec 9 01:16:13.223247510 (81h47m0.920815826s) [noise=true , address=81.171.14.70:4045 , piece_id=FD2J6MOI2FZWTDD4JCGAEQHGK6AUQNEQWLQ7PRGCJIV7L4VEXWDQ, seg_pos={0 0}, upload=71769]: Long tail cancellation due to optimal threshold Dec 9 01:16:13.223590229 (81h47m0.921158566s) [noise=true , address=81.171.14.70:4033 , piece_id=ETKLQ4F4LJ2Y75WZS4V6D3BP3SGX43ENIJZES7EVORZFHGBBLNWA, seg_pos={0 0}, upload=71769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 9 01:16:13.223600700 (81h47m0.921169005s) [noise=true , address=81.171.14.70:4033 , piece_id=ETKLQ4F4LJ2Y75WZS4V6D3BP3SGX43ENIJZES7EVORZFHGBBLNWA, seg_pos={0 0}, upload=71769]: Long tail cancellation due to optimal threshold Dec 9 01:16:13.223618643 (81h47m0.921186959s) [seg_pos={0 0}, upload=71769]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:16:13.223676281 (81h47m0.921244597s) [upload=71769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:16:13.223680419 (81h47m0.921248725s) [upload=71769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:16:13.372835035 (81h47m1.070403351s) [upload=71769]: Done uploading segments... Dec 9 01:16:13.372850073 (81h47m1.070418389s) [upload=71769]: Upload finished. err: <nil> Duration: 35.609479393s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 709.570853ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:16:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 709.611478ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 14 out of 60 is failed (threshold=3) [.......X...XX.....XX..X...X.....X.X.......X...X.....X.X.X...] duration: 6.071332866s trace: https://jaeger.dev.storj.io/trace/55aa152d476b0b56 FAIL (2025-12-09 01:16:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 201.147:4022 , piece_id=IF674FRHMSXK5JRJRX5KUMRFKY6QYHUJH6XAMSH7BNL2SADYBWUA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.385213444 (81h43m38.082781749s) [noise=true , address=185.167.204.205:4018 , piece_id=KJN2VEYGHCJNGMWMVREZHYVOXQJTXSCM7HZTBKZIGTWETJAFOKRQ, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.385584377 (81h43m38.083152683s) [noise=true , address=185.167.204.206:4045 , piece_id=6VGJABJY2PJWUSCWISP7LAROUBQDEZVGPGGBBGSWZ3BLITKLU3VA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.385948387 (81h43m38.083516713s) [noise=true , address=81.171.14.70:4040 , piece_id=OPO5GRTJJAMIXD4CAE2JV25225K4AY4IL6UR7AZTNKQ4ENOYC7WQ, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.386304823 (81h43m38.083873129s) [noise=true , address=89.149.201.147:4011 , piece_id=7UU5O2P7YGFLN564GQMSLYMU23XR5RMHCBMCSGRFHYYTGAW66REA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.386676267 (81h43m38.084244592s) [noise=true , address=81.171.14.70:4025 , piece_id=BEEZY6MMEIB4N5S5SBTQ7UW5NWRYWY73XXKJLR5MSZERX4BHKXOA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387010922 (81h43m38.084579238s) [noise=true , address=89.149.201.147:4007 , piece_id=SLM6EHIS35EBUJWKWKQNYKHPOP3NX3PDU6COP2WYGPV2SVL4U7SQ, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387358381 (81h43m38.084926687s) [noise=true , address=185.167.204.206:4037 , piece_id=RF42XFE6DKBVQAUKG4B4ATFY2H73YH3IG7RE7GABXITY46JEYA5Q, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387396963 (81h43m38.084965269s) [noise=true , address=185.167.204.206:4052 , piece_id=A3GXY3TAMCQV6RYTR5SQJ2V2KX7L5DWEAWPAVEDQK2JJJHA6355A, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387771183 (81h43m38.085339488s) [noise=true , address=185.167.204.205:4012 , piece_id=S6BTMFZY7GHBPP64BNDANKJUM22OWFHHVG7TSOKQQH3FFHMR4VZA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.388070921 (81h43m38.085639228s) [noise=true , address=89.149.201.147:4024 , piece_id=3GOHPUWK3J6DHXJUUNTDAPYHA474RNBLMAUU6HKVILZMHGU4CPRA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.878016887 (81h43m38.575585192s) [upload=71718]: Next returned nil segment Dec 9 01:12:50.878025393 (81h43m38.575593709s) [upload=71718]: Waiting for error group managing segments... Dec 9 01:12:51.068002823 (81h43m38.765571139s) [noise=true , address=89.149.201.147:4007 , piece_id=SLM6EHIS35EBUJWKWKQNYKHPOP3NX3PDU6COP2WYGPV2SVL4U7SQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.083282238 (81h43m38.780850543s) [noise=true , address=185.167.204.206:4037 , piece_id=RF42XFE6DKBVQAUKG4B4ATFY2H73YH3IG7RE7GABXITY46JEYA5Q, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.101125933 (81h43m38.798694249s) [noise=true , address=89.149.201.147:4009 , piece_id=M6GFXCGAMASIJW3CV75EURMCXAFR5EZVVTERLWMQ4GBN73MDBPFA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.108040971 (81h43m38.805609277s) [noise=true , address=185.167.204.205:4012 , piece_id=S6BTMFZY7GHBPP64BNDANKJUM22OWFHHVG7TSOKQQH3FFHMR4VZA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.145223114 (81h43m38.84279142s) [noise=true , address=89.149.201.147:4017 , piece_id=TAUT4QHTBR7HO6VYKKHB5YEUO44XIAC24YCEYKDTOKZPLQWJVCZA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.154067667 (81h43m38.851635983s) [noise=true , address=185.167.204.205:4011 , piece_id=G2XW7PF4M7ECAERK2EKERDFAFJSZJJ4LB7PT5ACMLH3R5TA66OMQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.396698113 (81h43m39.094266429s) [noise=true , address=185.167.204.206:4070 , piece_id=BXRL2JXHIEXAT3UPYD4XMIFIV3EQ7YIEDGFH4UHCYRMOA5DDUMXQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.408292734 (81h43m39.105861039s) [noise=true , address=185.167.204.205:4016 , piece_id=OYA6ZESX3PE3IDBV62WXMJ2AJSWCUWGSMG37UNSPW3GFZGEET3ZA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.461426216 (81h43m39.158994522s) [noise=true , address=81.171.14.70:4040 , piece_id=OPO5GRTJJAMIXD4CAE2JV25225K4AY4IL6UR7AZTNKQ4ENOYC7WQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.556765416 (81h43m39.254333722s) [noise=true , address=89.149.201.147:4005 , piece_id=MQFPZ2SFYS4SBJEKNEFKCP36AVGXT4DEXZJULOGIQSCMEL443EDA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.585384632 (81h43m39.282952948s) [noise=true , address=185.167.204.206:4045 , piece_id=6VGJABJY2PJWUSCWISP7LAROUBQDEZVGPGGBBGSWZ3BLITKLU3VA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.597392695 (81h43m39.294961011s) [noise=true , address=185.167.204.205:4005 , piece_id=GLFHIVZSOUKRVHPKNJ36Y6UY7W3GV6ZE7O73NSDQTEZTF7J7GODA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.614140743 (81h43m39.311709059s) [noise=true , address=89.149.201.147:4022 , piece_id=IF674FRHMSXK5JRJRX5KUMRFKY6QYHUJH6XAMSH7BNL2SADYBWUA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.670996619 (81h43m39.368564926s) [noise=true , address=185.167.204.205:4002 , piece_id=BC5U2W5AU2YCERC2HHXRHGQLHIBZGPXLPIPI23A43DTY5E6QWLGA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.781046552 (81h43m39.478614858s) [noise=true , address=185.167.204.205:4006 , piece_id=OPFYYA472OFJZRXVA5ZUUUQU3NBH6H6Q6P72VCA2SQFO62GEAIKQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.852222730 (81h43m39.549791046s) [noise=true , address=89.149.201.147:4011 , piece_id=7UU5O2P7YGFLN564GQMSLYMU23XR5RMHCBMCSGRFHYYTGAW66REA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.858271942 (81h43m39.555840249s) [noise=true , address=81.171.14.70:4025 , piece_id=BEEZY6MMEIB4N5S5SBTQ7UW5NWRYWY73XXKJLR5MSZERX4BHKXOA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.880329700 (81h43m39.577898046s) [noise=true , address=185.167.204.205:4033 , piece_id=IZPRZT2MU55T35UUJOTMOOEGMU5VBKNCZ5M5TGNCH36AGKHS3I5Q, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.920106870 (81h43m39.617675186s) [noise=true , address=81.171.14.70:4039 , piece_id=3CW2XRBEOPRTAVDOMXGAV2YTYX5QEN4LL2SCHJ7QKN3GDXOJYS3A, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.090238325 (81h43m39.787806641s) [noise=true , address=185.167.204.206:4058 , piece_id=ROW6CMCMISJH2ECXJH3BZS7E6RFN4CAQD5AX55FDFZMDMDAWVPYQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.190060481 (81h43m39.887628797s) [noise=true , address=89.149.201.147:4024 , piece_id=3GOHPUWK3J6DHXJUUNTDAPYHA474RNBLMAUU6HKVILZMHGU4CPRA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.447863852 (81h43m40.145432168s) [noise=true , address=81.171.14.70:4046 , piece_id=HPSANBEG4LVA4CQU2FTOT3EMIGYNLC6PUOW5F7M7JEJIO2P3ZI5Q, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.743172328 (81h43m40.440740644s) [noise=true , address=185.167.204.205:4030 , piece_id=WHFHRLZ32ZG763A7PVDJFF66QFTZWRRJDMWYXVBI7JFUISP2MSMQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.841645182 (81h43m40.539213488s) [noise=true , address=185.167.204.206:4052 , piece_id=A3GXY3TAMCQV6RYTR5SQJ2V2KX7L5DWEAWPAVEDQK2JJJHA6355A, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.871944208 (81h43m40.569512524s) [noise=true , address=185.167.204.205:4018 , piece_id=KJN2VEYGHCJNGMWMVREZHYVOXQJTXSCM7HZTBKZIGTWETJAFOKRQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.992610002 (81h43m40.690178318s) [noise=true , address=81.171.14.70:4047 , piece_id=Z7L2MXQQGI6YTKMWRKAPB63Z72X6LRMTRUZB66PUYOZVENY4EXEA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.992623887 (81h43m40.690192193s) [seg_pos={0 0}, upload=71718]: Segment reached optimal threshold of 26 pieces. Dec 9 01:12:52.992754642 (81h43m40.690322948s) [noise=true , address=81.171.14.70:4029 , piece_id=RLIF2LUWZ37QYE3MRJ73OB3WTN6HSD6ESBARTKSO3ECHCFOZK6EQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to): protocol: expected piece hash; context canceled; EOF Dec 9 01:12:52.992759992 (81h43m40.690328297s) [noise=true , address=81.171.14.70:4029 , piece_id=RLIF2LUWZ37QYE3MRJ73OB3WTN6HSD6ESBARTKSO3ECHCFOZK6EQ, seg_pos={0 0}, upload=71718]: Long tail cancellation due to optimal threshold Dec 9 01:12:52.992901035 (81h43m40.690469351s) [noise=true , address=81.171.14.70:4031 , piece_id=ULW72EQDNI3B4PU3HADKLQPNHMTHF3VMH6UBFO73DCO22NQGDOKQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 9 01:12:52.992906495 (81h43m40.690474801s) [noise=true , address=81.171.14.70:4031 , piece_id=ULW72EQDNI3B4PU3HADKLQPNHMTHF3VMH6UBFO73DCO22NQGDOKQ, seg_pos={0 0}, upload=71718]: Long tail cancellation due to optimal threshold Dec 9 01:12:52.992918939 (81h43m40.690487255s) [seg_pos={0 0}, upload=71718]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:12:52.992974452 (81h43m40.690542758s) [upload=71718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:12:52.992978059 (81h43m40.690546375s) [upload=71718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:12:53.141659621 (81h43m40.839227967s) [upload=71718]: Done uploading segments... Dec 9 01:12:53.141690719 (81h43m40.839259075s) [upload=71718]: Upload finished. err: <nil> Duration: 6.071352733s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 465.641789ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:15:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 465.661225ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 14 out of 60 is failed (threshold=3) [........X...XX.....XX..X...X.....X.X.......X...X.....X.X.X..] duration: 13.370466011s trace: https://jaeger.dev.storj.io/trace/55aa152d476b0b56 FAIL (2025-12-09 01:14:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 201.147:4022 , piece_id=IF674FRHMSXK5JRJRX5KUMRFKY6QYHUJH6XAMSH7BNL2SADYBWUA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.385213444 (81h43m38.082781749s) [noise=true , address=185.167.204.205:4018 , piece_id=KJN2VEYGHCJNGMWMVREZHYVOXQJTXSCM7HZTBKZIGTWETJAFOKRQ, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.385584377 (81h43m38.083152683s) [noise=true , address=185.167.204.206:4045 , piece_id=6VGJABJY2PJWUSCWISP7LAROUBQDEZVGPGGBBGSWZ3BLITKLU3VA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.385948387 (81h43m38.083516713s) [noise=true , address=81.171.14.70:4040 , piece_id=OPO5GRTJJAMIXD4CAE2JV25225K4AY4IL6UR7AZTNKQ4ENOYC7WQ, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.386304823 (81h43m38.083873129s) [noise=true , address=89.149.201.147:4011 , piece_id=7UU5O2P7YGFLN564GQMSLYMU23XR5RMHCBMCSGRFHYYTGAW66REA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.386676267 (81h43m38.084244592s) [noise=true , address=81.171.14.70:4025 , piece_id=BEEZY6MMEIB4N5S5SBTQ7UW5NWRYWY73XXKJLR5MSZERX4BHKXOA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387010922 (81h43m38.084579238s) [noise=true , address=89.149.201.147:4007 , piece_id=SLM6EHIS35EBUJWKWKQNYKHPOP3NX3PDU6COP2WYGPV2SVL4U7SQ, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387358381 (81h43m38.084926687s) [noise=true , address=185.167.204.206:4037 , piece_id=RF42XFE6DKBVQAUKG4B4ATFY2H73YH3IG7RE7GABXITY46JEYA5Q, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387396963 (81h43m38.084965269s) [noise=true , address=185.167.204.206:4052 , piece_id=A3GXY3TAMCQV6RYTR5SQJ2V2KX7L5DWEAWPAVEDQK2JJJHA6355A, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387771183 (81h43m38.085339488s) [noise=true , address=185.167.204.205:4012 , piece_id=S6BTMFZY7GHBPP64BNDANKJUM22OWFHHVG7TSOKQQH3FFHMR4VZA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.388070921 (81h43m38.085639228s) [noise=true , address=89.149.201.147:4024 , piece_id=3GOHPUWK3J6DHXJUUNTDAPYHA474RNBLMAUU6HKVILZMHGU4CPRA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.878016887 (81h43m38.575585192s) [upload=71718]: Next returned nil segment Dec 9 01:12:50.878025393 (81h43m38.575593709s) [upload=71718]: Waiting for error group managing segments... Dec 9 01:12:51.068002823 (81h43m38.765571139s) [noise=true , address=89.149.201.147:4007 , piece_id=SLM6EHIS35EBUJWKWKQNYKHPOP3NX3PDU6COP2WYGPV2SVL4U7SQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.083282238 (81h43m38.780850543s) [noise=true , address=185.167.204.206:4037 , piece_id=RF42XFE6DKBVQAUKG4B4ATFY2H73YH3IG7RE7GABXITY46JEYA5Q, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.101125933 (81h43m38.798694249s) [noise=true , address=89.149.201.147:4009 , piece_id=M6GFXCGAMASIJW3CV75EURMCXAFR5EZVVTERLWMQ4GBN73MDBPFA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.108040971 (81h43m38.805609277s) [noise=true , address=185.167.204.205:4012 , piece_id=S6BTMFZY7GHBPP64BNDANKJUM22OWFHHVG7TSOKQQH3FFHMR4VZA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.145223114 (81h43m38.84279142s) [noise=true , address=89.149.201.147:4017 , piece_id=TAUT4QHTBR7HO6VYKKHB5YEUO44XIAC24YCEYKDTOKZPLQWJVCZA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.154067667 (81h43m38.851635983s) [noise=true , address=185.167.204.205:4011 , piece_id=G2XW7PF4M7ECAERK2EKERDFAFJSZJJ4LB7PT5ACMLH3R5TA66OMQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.396698113 (81h43m39.094266429s) [noise=true , address=185.167.204.206:4070 , piece_id=BXRL2JXHIEXAT3UPYD4XMIFIV3EQ7YIEDGFH4UHCYRMOA5DDUMXQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.408292734 (81h43m39.105861039s) [noise=true , address=185.167.204.205:4016 , piece_id=OYA6ZESX3PE3IDBV62WXMJ2AJSWCUWGSMG37UNSPW3GFZGEET3ZA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.461426216 (81h43m39.158994522s) [noise=true , address=81.171.14.70:4040 , piece_id=OPO5GRTJJAMIXD4CAE2JV25225K4AY4IL6UR7AZTNKQ4ENOYC7WQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.556765416 (81h43m39.254333722s) [noise=true , address=89.149.201.147:4005 , piece_id=MQFPZ2SFYS4SBJEKNEFKCP36AVGXT4DEXZJULOGIQSCMEL443EDA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.585384632 (81h43m39.282952948s) [noise=true , address=185.167.204.206:4045 , piece_id=6VGJABJY2PJWUSCWISP7LAROUBQDEZVGPGGBBGSWZ3BLITKLU3VA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.597392695 (81h43m39.294961011s) [noise=true , address=185.167.204.205:4005 , piece_id=GLFHIVZSOUKRVHPKNJ36Y6UY7W3GV6ZE7O73NSDQTEZTF7J7GODA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.614140743 (81h43m39.311709059s) [noise=true , address=89.149.201.147:4022 , piece_id=IF674FRHMSXK5JRJRX5KUMRFKY6QYHUJH6XAMSH7BNL2SADYBWUA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.670996619 (81h43m39.368564926s) [noise=true , address=185.167.204.205:4002 , piece_id=BC5U2W5AU2YCERC2HHXRHGQLHIBZGPXLPIPI23A43DTY5E6QWLGA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.781046552 (81h43m39.478614858s) [noise=true , address=185.167.204.205:4006 , piece_id=OPFYYA472OFJZRXVA5ZUUUQU3NBH6H6Q6P72VCA2SQFO62GEAIKQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.852222730 (81h43m39.549791046s) [noise=true , address=89.149.201.147:4011 , piece_id=7UU5O2P7YGFLN564GQMSLYMU23XR5RMHCBMCSGRFHYYTGAW66REA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.858271942 (81h43m39.555840249s) [noise=true , address=81.171.14.70:4025 , piece_id=BEEZY6MMEIB4N5S5SBTQ7UW5NWRYWY73XXKJLR5MSZERX4BHKXOA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.880329700 (81h43m39.577898046s) [noise=true , address=185.167.204.205:4033 , piece_id=IZPRZT2MU55T35UUJOTMOOEGMU5VBKNCZ5M5TGNCH36AGKHS3I5Q, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.920106870 (81h43m39.617675186s) [noise=true , address=81.171.14.70:4039 , piece_id=3CW2XRBEOPRTAVDOMXGAV2YTYX5QEN4LL2SCHJ7QKN3GDXOJYS3A, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.090238325 (81h43m39.787806641s) [noise=true , address=185.167.204.206:4058 , piece_id=ROW6CMCMISJH2ECXJH3BZS7E6RFN4CAQD5AX55FDFZMDMDAWVPYQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.190060481 (81h43m39.887628797s) [noise=true , address=89.149.201.147:4024 , piece_id=3GOHPUWK3J6DHXJUUNTDAPYHA474RNBLMAUU6HKVILZMHGU4CPRA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.447863852 (81h43m40.145432168s) [noise=true , address=81.171.14.70:4046 , piece_id=HPSANBEG4LVA4CQU2FTOT3EMIGYNLC6PUOW5F7M7JEJIO2P3ZI5Q, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.743172328 (81h43m40.440740644s) [noise=true , address=185.167.204.205:4030 , piece_id=WHFHRLZ32ZG763A7PVDJFF66QFTZWRRJDMWYXVBI7JFUISP2MSMQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.841645182 (81h43m40.539213488s) [noise=true , address=185.167.204.206:4052 , piece_id=A3GXY3TAMCQV6RYTR5SQJ2V2KX7L5DWEAWPAVEDQK2JJJHA6355A, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.871944208 (81h43m40.569512524s) [noise=true , address=185.167.204.205:4018 , piece_id=KJN2VEYGHCJNGMWMVREZHYVOXQJTXSCM7HZTBKZIGTWETJAFOKRQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.992610002 (81h43m40.690178318s) [noise=true , address=81.171.14.70:4047 , piece_id=Z7L2MXQQGI6YTKMWRKAPB63Z72X6LRMTRUZB66PUYOZVENY4EXEA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.992623887 (81h43m40.690192193s) [seg_pos={0 0}, upload=71718]: Segment reached optimal threshold of 26 pieces. Dec 9 01:12:52.992754642 (81h43m40.690322948s) [noise=true , address=81.171.14.70:4029 , piece_id=RLIF2LUWZ37QYE3MRJ73OB3WTN6HSD6ESBARTKSO3ECHCFOZK6EQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to): protocol: expected piece hash; context canceled; EOF Dec 9 01:12:52.992759992 (81h43m40.690328297s) [noise=true , address=81.171.14.70:4029 , piece_id=RLIF2LUWZ37QYE3MRJ73OB3WTN6HSD6ESBARTKSO3ECHCFOZK6EQ, seg_pos={0 0}, upload=71718]: Long tail cancellation due to optimal threshold Dec 9 01:12:52.992901035 (81h43m40.690469351s) [noise=true , address=81.171.14.70:4031 , piece_id=ULW72EQDNI3B4PU3HADKLQPNHMTHF3VMH6UBFO73DCO22NQGDOKQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 9 01:12:52.992906495 (81h43m40.690474801s) [noise=true , address=81.171.14.70:4031 , piece_id=ULW72EQDNI3B4PU3HADKLQPNHMTHF3VMH6UBFO73DCO22NQGDOKQ, seg_pos={0 0}, upload=71718]: Long tail cancellation due to optimal threshold Dec 9 01:12:52.992918939 (81h43m40.690487255s) [seg_pos={0 0}, upload=71718]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:12:52.992974452 (81h43m40.690542758s) [upload=71718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:12:52.992978059 (81h43m40.690546375s) [upload=71718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:12:53.141659621 (81h43m40.839227967s) [upload=71718]: Done uploading segments... Dec 9 01:12:53.141690719 (81h43m40.839259075s) [upload=71718]: Upload finished. err: <nil> Duration: 13.37048755s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 434.135352ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:14:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.158144ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 15 out of 60 is failed (threshold=3) [X........X...XX.....XX..X...X.....X.X.......X...X.....X.X.X.] duration: 5.484081382s trace: https://jaeger.dev.storj.io/trace/55aa152d476b0b56 FAIL (2025-12-09 01:13:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 201.147:4022 , piece_id=IF674FRHMSXK5JRJRX5KUMRFKY6QYHUJH6XAMSH7BNL2SADYBWUA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.385213444 (81h43m38.082781749s) [noise=true , address=185.167.204.205:4018 , piece_id=KJN2VEYGHCJNGMWMVREZHYVOXQJTXSCM7HZTBKZIGTWETJAFOKRQ, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.385584377 (81h43m38.083152683s) [noise=true , address=185.167.204.206:4045 , piece_id=6VGJABJY2PJWUSCWISP7LAROUBQDEZVGPGGBBGSWZ3BLITKLU3VA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.385948387 (81h43m38.083516713s) [noise=true , address=81.171.14.70:4040 , piece_id=OPO5GRTJJAMIXD4CAE2JV25225K4AY4IL6UR7AZTNKQ4ENOYC7WQ, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.386304823 (81h43m38.083873129s) [noise=true , address=89.149.201.147:4011 , piece_id=7UU5O2P7YGFLN564GQMSLYMU23XR5RMHCBMCSGRFHYYTGAW66REA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.386676267 (81h43m38.084244592s) [noise=true , address=81.171.14.70:4025 , piece_id=BEEZY6MMEIB4N5S5SBTQ7UW5NWRYWY73XXKJLR5MSZERX4BHKXOA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387010922 (81h43m38.084579238s) [noise=true , address=89.149.201.147:4007 , piece_id=SLM6EHIS35EBUJWKWKQNYKHPOP3NX3PDU6COP2WYGPV2SVL4U7SQ, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387358381 (81h43m38.084926687s) [noise=true , address=185.167.204.206:4037 , piece_id=RF42XFE6DKBVQAUKG4B4ATFY2H73YH3IG7RE7GABXITY46JEYA5Q, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387396963 (81h43m38.084965269s) [noise=true , address=185.167.204.206:4052 , piece_id=A3GXY3TAMCQV6RYTR5SQJ2V2KX7L5DWEAWPAVEDQK2JJJHA6355A, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387771183 (81h43m38.085339488s) [noise=true , address=185.167.204.205:4012 , piece_id=S6BTMFZY7GHBPP64BNDANKJUM22OWFHHVG7TSOKQQH3FFHMR4VZA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.388070921 (81h43m38.085639228s) [noise=true , address=89.149.201.147:4024 , piece_id=3GOHPUWK3J6DHXJUUNTDAPYHA474RNBLMAUU6HKVILZMHGU4CPRA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.878016887 (81h43m38.575585192s) [upload=71718]: Next returned nil segment Dec 9 01:12:50.878025393 (81h43m38.575593709s) [upload=71718]: Waiting for error group managing segments... Dec 9 01:12:51.068002823 (81h43m38.765571139s) [noise=true , address=89.149.201.147:4007 , piece_id=SLM6EHIS35EBUJWKWKQNYKHPOP3NX3PDU6COP2WYGPV2SVL4U7SQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.083282238 (81h43m38.780850543s) [noise=true , address=185.167.204.206:4037 , piece_id=RF42XFE6DKBVQAUKG4B4ATFY2H73YH3IG7RE7GABXITY46JEYA5Q, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.101125933 (81h43m38.798694249s) [noise=true , address=89.149.201.147:4009 , piece_id=M6GFXCGAMASIJW3CV75EURMCXAFR5EZVVTERLWMQ4GBN73MDBPFA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.108040971 (81h43m38.805609277s) [noise=true , address=185.167.204.205:4012 , piece_id=S6BTMFZY7GHBPP64BNDANKJUM22OWFHHVG7TSOKQQH3FFHMR4VZA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.145223114 (81h43m38.84279142s) [noise=true , address=89.149.201.147:4017 , piece_id=TAUT4QHTBR7HO6VYKKHB5YEUO44XIAC24YCEYKDTOKZPLQWJVCZA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.154067667 (81h43m38.851635983s) [noise=true , address=185.167.204.205:4011 , piece_id=G2XW7PF4M7ECAERK2EKERDFAFJSZJJ4LB7PT5ACMLH3R5TA66OMQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.396698113 (81h43m39.094266429s) [noise=true , address=185.167.204.206:4070 , piece_id=BXRL2JXHIEXAT3UPYD4XMIFIV3EQ7YIEDGFH4UHCYRMOA5DDUMXQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.408292734 (81h43m39.105861039s) [noise=true , address=185.167.204.205:4016 , piece_id=OYA6ZESX3PE3IDBV62WXMJ2AJSWCUWGSMG37UNSPW3GFZGEET3ZA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.461426216 (81h43m39.158994522s) [noise=true , address=81.171.14.70:4040 , piece_id=OPO5GRTJJAMIXD4CAE2JV25225K4AY4IL6UR7AZTNKQ4ENOYC7WQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.556765416 (81h43m39.254333722s) [noise=true , address=89.149.201.147:4005 , piece_id=MQFPZ2SFYS4SBJEKNEFKCP36AVGXT4DEXZJULOGIQSCMEL443EDA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.585384632 (81h43m39.282952948s) [noise=true , address=185.167.204.206:4045 , piece_id=6VGJABJY2PJWUSCWISP7LAROUBQDEZVGPGGBBGSWZ3BLITKLU3VA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.597392695 (81h43m39.294961011s) [noise=true , address=185.167.204.205:4005 , piece_id=GLFHIVZSOUKRVHPKNJ36Y6UY7W3GV6ZE7O73NSDQTEZTF7J7GODA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.614140743 (81h43m39.311709059s) [noise=true , address=89.149.201.147:4022 , piece_id=IF674FRHMSXK5JRJRX5KUMRFKY6QYHUJH6XAMSH7BNL2SADYBWUA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.670996619 (81h43m39.368564926s) [noise=true , address=185.167.204.205:4002 , piece_id=BC5U2W5AU2YCERC2HHXRHGQLHIBZGPXLPIPI23A43DTY5E6QWLGA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.781046552 (81h43m39.478614858s) [noise=true , address=185.167.204.205:4006 , piece_id=OPFYYA472OFJZRXVA5ZUUUQU3NBH6H6Q6P72VCA2SQFO62GEAIKQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.852222730 (81h43m39.549791046s) [noise=true , address=89.149.201.147:4011 , piece_id=7UU5O2P7YGFLN564GQMSLYMU23XR5RMHCBMCSGRFHYYTGAW66REA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.858271942 (81h43m39.555840249s) [noise=true , address=81.171.14.70:4025 , piece_id=BEEZY6MMEIB4N5S5SBTQ7UW5NWRYWY73XXKJLR5MSZERX4BHKXOA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.880329700 (81h43m39.577898046s) [noise=true , address=185.167.204.205:4033 , piece_id=IZPRZT2MU55T35UUJOTMOOEGMU5VBKNCZ5M5TGNCH36AGKHS3I5Q, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.920106870 (81h43m39.617675186s) [noise=true , address=81.171.14.70:4039 , piece_id=3CW2XRBEOPRTAVDOMXGAV2YTYX5QEN4LL2SCHJ7QKN3GDXOJYS3A, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.090238325 (81h43m39.787806641s) [noise=true , address=185.167.204.206:4058 , piece_id=ROW6CMCMISJH2ECXJH3BZS7E6RFN4CAQD5AX55FDFZMDMDAWVPYQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.190060481 (81h43m39.887628797s) [noise=true , address=89.149.201.147:4024 , piece_id=3GOHPUWK3J6DHXJUUNTDAPYHA474RNBLMAUU6HKVILZMHGU4CPRA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.447863852 (81h43m40.145432168s) [noise=true , address=81.171.14.70:4046 , piece_id=HPSANBEG4LVA4CQU2FTOT3EMIGYNLC6PUOW5F7M7JEJIO2P3ZI5Q, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.743172328 (81h43m40.440740644s) [noise=true , address=185.167.204.205:4030 , piece_id=WHFHRLZ32ZG763A7PVDJFF66QFTZWRRJDMWYXVBI7JFUISP2MSMQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.841645182 (81h43m40.539213488s) [noise=true , address=185.167.204.206:4052 , piece_id=A3GXY3TAMCQV6RYTR5SQJ2V2KX7L5DWEAWPAVEDQK2JJJHA6355A, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.871944208 (81h43m40.569512524s) [noise=true , address=185.167.204.205:4018 , piece_id=KJN2VEYGHCJNGMWMVREZHYVOXQJTXSCM7HZTBKZIGTWETJAFOKRQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.992610002 (81h43m40.690178318s) [noise=true , address=81.171.14.70:4047 , piece_id=Z7L2MXQQGI6YTKMWRKAPB63Z72X6LRMTRUZB66PUYOZVENY4EXEA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.992623887 (81h43m40.690192193s) [seg_pos={0 0}, upload=71718]: Segment reached optimal threshold of 26 pieces. Dec 9 01:12:52.992754642 (81h43m40.690322948s) [noise=true , address=81.171.14.70:4029 , piece_id=RLIF2LUWZ37QYE3MRJ73OB3WTN6HSD6ESBARTKSO3ECHCFOZK6EQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to): protocol: expected piece hash; context canceled; EOF Dec 9 01:12:52.992759992 (81h43m40.690328297s) [noise=true , address=81.171.14.70:4029 , piece_id=RLIF2LUWZ37QYE3MRJ73OB3WTN6HSD6ESBARTKSO3ECHCFOZK6EQ, seg_pos={0 0}, upload=71718]: Long tail cancellation due to optimal threshold Dec 9 01:12:52.992901035 (81h43m40.690469351s) [noise=true , address=81.171.14.70:4031 , piece_id=ULW72EQDNI3B4PU3HADKLQPNHMTHF3VMH6UBFO73DCO22NQGDOKQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 9 01:12:52.992906495 (81h43m40.690474801s) [noise=true , address=81.171.14.70:4031 , piece_id=ULW72EQDNI3B4PU3HADKLQPNHMTHF3VMH6UBFO73DCO22NQGDOKQ, seg_pos={0 0}, upload=71718]: Long tail cancellation due to optimal threshold Dec 9 01:12:52.992918939 (81h43m40.690487255s) [seg_pos={0 0}, upload=71718]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:12:52.992974452 (81h43m40.690542758s) [upload=71718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:12:52.992978059 (81h43m40.690546375s) [upload=71718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:12:53.141659621 (81h43m40.839227967s) [upload=71718]: Done uploading segments... Dec 9 01:12:53.141690719 (81h43m40.839259075s) [upload=71718]: Upload finished. err: <nil> Duration: 5.484093466s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 15 out of 60 is failed (threshold=3) [.X........X...XX.....XX..X...X.....X.X.......X...X.....X.X.X] duration: 1m0.003825523s trace: https://jaeger.dev.storj.io/trace/55aa152d476b0b56 FAIL (2025-12-09 01:13:49 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 201.147:4022 , piece_id=IF674FRHMSXK5JRJRX5KUMRFKY6QYHUJH6XAMSH7BNL2SADYBWUA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.385213444 (81h43m38.082781749s) [noise=true , address=185.167.204.205:4018 , piece_id=KJN2VEYGHCJNGMWMVREZHYVOXQJTXSCM7HZTBKZIGTWETJAFOKRQ, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.385584377 (81h43m38.083152683s) [noise=true , address=185.167.204.206:4045 , piece_id=6VGJABJY2PJWUSCWISP7LAROUBQDEZVGPGGBBGSWZ3BLITKLU3VA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.385948387 (81h43m38.083516713s) [noise=true , address=81.171.14.70:4040 , piece_id=OPO5GRTJJAMIXD4CAE2JV25225K4AY4IL6UR7AZTNKQ4ENOYC7WQ, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.386304823 (81h43m38.083873129s) [noise=true , address=89.149.201.147:4011 , piece_id=7UU5O2P7YGFLN564GQMSLYMU23XR5RMHCBMCSGRFHYYTGAW66REA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.386676267 (81h43m38.084244592s) [noise=true , address=81.171.14.70:4025 , piece_id=BEEZY6MMEIB4N5S5SBTQ7UW5NWRYWY73XXKJLR5MSZERX4BHKXOA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387010922 (81h43m38.084579238s) [noise=true , address=89.149.201.147:4007 , piece_id=SLM6EHIS35EBUJWKWKQNYKHPOP3NX3PDU6COP2WYGPV2SVL4U7SQ, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387358381 (81h43m38.084926687s) [noise=true , address=185.167.204.206:4037 , piece_id=RF42XFE6DKBVQAUKG4B4ATFY2H73YH3IG7RE7GABXITY46JEYA5Q, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387396963 (81h43m38.084965269s) [noise=true , address=185.167.204.206:4052 , piece_id=A3GXY3TAMCQV6RYTR5SQJ2V2KX7L5DWEAWPAVEDQK2JJJHA6355A, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.387771183 (81h43m38.085339488s) [noise=true , address=185.167.204.205:4012 , piece_id=S6BTMFZY7GHBPP64BNDANKJUM22OWFHHVG7TSOKQQH3FFHMR4VZA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.388070921 (81h43m38.085639228s) [noise=true , address=89.149.201.147:4024 , piece_id=3GOHPUWK3J6DHXJUUNTDAPYHA474RNBLMAUU6HKVILZMHGU4CPRA, seg_pos={0 0}, upload=71718]: Uploading piece... Dec 9 01:12:50.878016887 (81h43m38.575585192s) [upload=71718]: Next returned nil segment Dec 9 01:12:50.878025393 (81h43m38.575593709s) [upload=71718]: Waiting for error group managing segments... Dec 9 01:12:51.068002823 (81h43m38.765571139s) [noise=true , address=89.149.201.147:4007 , piece_id=SLM6EHIS35EBUJWKWKQNYKHPOP3NX3PDU6COP2WYGPV2SVL4U7SQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.083282238 (81h43m38.780850543s) [noise=true , address=185.167.204.206:4037 , piece_id=RF42XFE6DKBVQAUKG4B4ATFY2H73YH3IG7RE7GABXITY46JEYA5Q, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.101125933 (81h43m38.798694249s) [noise=true , address=89.149.201.147:4009 , piece_id=M6GFXCGAMASIJW3CV75EURMCXAFR5EZVVTERLWMQ4GBN73MDBPFA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.108040971 (81h43m38.805609277s) [noise=true , address=185.167.204.205:4012 , piece_id=S6BTMFZY7GHBPP64BNDANKJUM22OWFHHVG7TSOKQQH3FFHMR4VZA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.145223114 (81h43m38.84279142s) [noise=true , address=89.149.201.147:4017 , piece_id=TAUT4QHTBR7HO6VYKKHB5YEUO44XIAC24YCEYKDTOKZPLQWJVCZA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.154067667 (81h43m38.851635983s) [noise=true , address=185.167.204.205:4011 , piece_id=G2XW7PF4M7ECAERK2EKERDFAFJSZJJ4LB7PT5ACMLH3R5TA66OMQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.396698113 (81h43m39.094266429s) [noise=true , address=185.167.204.206:4070 , piece_id=BXRL2JXHIEXAT3UPYD4XMIFIV3EQ7YIEDGFH4UHCYRMOA5DDUMXQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.408292734 (81h43m39.105861039s) [noise=true , address=185.167.204.205:4016 , piece_id=OYA6ZESX3PE3IDBV62WXMJ2AJSWCUWGSMG37UNSPW3GFZGEET3ZA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.461426216 (81h43m39.158994522s) [noise=true , address=81.171.14.70:4040 , piece_id=OPO5GRTJJAMIXD4CAE2JV25225K4AY4IL6UR7AZTNKQ4ENOYC7WQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.556765416 (81h43m39.254333722s) [noise=true , address=89.149.201.147:4005 , piece_id=MQFPZ2SFYS4SBJEKNEFKCP36AVGXT4DEXZJULOGIQSCMEL443EDA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.585384632 (81h43m39.282952948s) [noise=true , address=185.167.204.206:4045 , piece_id=6VGJABJY2PJWUSCWISP7LAROUBQDEZVGPGGBBGSWZ3BLITKLU3VA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.597392695 (81h43m39.294961011s) [noise=true , address=185.167.204.205:4005 , piece_id=GLFHIVZSOUKRVHPKNJ36Y6UY7W3GV6ZE7O73NSDQTEZTF7J7GODA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.614140743 (81h43m39.311709059s) [noise=true , address=89.149.201.147:4022 , piece_id=IF674FRHMSXK5JRJRX5KUMRFKY6QYHUJH6XAMSH7BNL2SADYBWUA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.670996619 (81h43m39.368564926s) [noise=true , address=185.167.204.205:4002 , piece_id=BC5U2W5AU2YCERC2HHXRHGQLHIBZGPXLPIPI23A43DTY5E6QWLGA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.781046552 (81h43m39.478614858s) [noise=true , address=185.167.204.205:4006 , piece_id=OPFYYA472OFJZRXVA5ZUUUQU3NBH6H6Q6P72VCA2SQFO62GEAIKQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.852222730 (81h43m39.549791046s) [noise=true , address=89.149.201.147:4011 , piece_id=7UU5O2P7YGFLN564GQMSLYMU23XR5RMHCBMCSGRFHYYTGAW66REA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.858271942 (81h43m39.555840249s) [noise=true , address=81.171.14.70:4025 , piece_id=BEEZY6MMEIB4N5S5SBTQ7UW5NWRYWY73XXKJLR5MSZERX4BHKXOA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.880329700 (81h43m39.577898046s) [noise=true , address=185.167.204.205:4033 , piece_id=IZPRZT2MU55T35UUJOTMOOEGMU5VBKNCZ5M5TGNCH36AGKHS3I5Q, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:51.920106870 (81h43m39.617675186s) [noise=true , address=81.171.14.70:4039 , piece_id=3CW2XRBEOPRTAVDOMXGAV2YTYX5QEN4LL2SCHJ7QKN3GDXOJYS3A, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.090238325 (81h43m39.787806641s) [noise=true , address=185.167.204.206:4058 , piece_id=ROW6CMCMISJH2ECXJH3BZS7E6RFN4CAQD5AX55FDFZMDMDAWVPYQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.190060481 (81h43m39.887628797s) [noise=true , address=89.149.201.147:4024 , piece_id=3GOHPUWK3J6DHXJUUNTDAPYHA474RNBLMAUU6HKVILZMHGU4CPRA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.447863852 (81h43m40.145432168s) [noise=true , address=81.171.14.70:4046 , piece_id=HPSANBEG4LVA4CQU2FTOT3EMIGYNLC6PUOW5F7M7JEJIO2P3ZI5Q, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.743172328 (81h43m40.440740644s) [noise=true , address=185.167.204.205:4030 , piece_id=WHFHRLZ32ZG763A7PVDJFF66QFTZWRRJDMWYXVBI7JFUISP2MSMQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.841645182 (81h43m40.539213488s) [noise=true , address=185.167.204.206:4052 , piece_id=A3GXY3TAMCQV6RYTR5SQJ2V2KX7L5DWEAWPAVEDQK2JJJHA6355A, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.871944208 (81h43m40.569512524s) [noise=true , address=185.167.204.205:4018 , piece_id=KJN2VEYGHCJNGMWMVREZHYVOXQJTXSCM7HZTBKZIGTWETJAFOKRQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.992610002 (81h43m40.690178318s) [noise=true , address=81.171.14.70:4047 , piece_id=Z7L2MXQQGI6YTKMWRKAPB63Z72X6LRMTRUZB66PUYOZVENY4EXEA, seg_pos={0 0}, upload=71718]: Done uploading piece. err: <nil> Dec 9 01:12:52.992623887 (81h43m40.690192193s) [seg_pos={0 0}, upload=71718]: Segment reached optimal threshold of 26 pieces. Dec 9 01:12:52.992754642 (81h43m40.690322948s) [noise=true , address=81.171.14.70:4029 , piece_id=RLIF2LUWZ37QYE3MRJ73OB3WTN6HSD6ESBARTKSO3ECHCFOZK6EQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to): protocol: expected piece hash; context canceled; EOF Dec 9 01:12:52.992759992 (81h43m40.690328297s) [noise=true , address=81.171.14.70:4029 , piece_id=RLIF2LUWZ37QYE3MRJ73OB3WTN6HSD6ESBARTKSO3ECHCFOZK6EQ, seg_pos={0 0}, upload=71718]: Long tail cancellation due to optimal threshold Dec 9 01:12:52.992901035 (81h43m40.690469351s) [noise=true , address=81.171.14.70:4031 , piece_id=ULW72EQDNI3B4PU3HADKLQPNHMTHF3VMH6UBFO73DCO22NQGDOKQ, seg_pos={0 0}, upload=71718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 9 01:12:52.992906495 (81h43m40.690474801s) [noise=true , address=81.171.14.70:4031 , piece_id=ULW72EQDNI3B4PU3HADKLQPNHMTHF3VMH6UBFO73DCO22NQGDOKQ, seg_pos={0 0}, upload=71718]: Long tail cancellation due to optimal threshold Dec 9 01:12:52.992918939 (81h43m40.690487255s) [seg_pos={0 0}, upload=71718]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:12:52.992974452 (81h43m40.690542758s) [upload=71718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:12:52.992978059 (81h43m40.690546375s) [upload=71718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:12:53.141659621 (81h43m40.839227967s) [upload=71718]: Done uploading segments... Dec 9 01:12:53.141690719 (81h43m40.839259075s) [upload=71718]: Upload finished. err: <nil> Duration: 1m0.003842755s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 504.838687ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:13:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 504.860026ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 441.346172ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:12:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.377961ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 468.354537ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:12:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 468.370907ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 14 out of 60 is failed (threshold=3) [..X........X...XX.....XX..X...X.....X.X.......X...X.....X.X.] duration: 4.330357557s trace: https://jaeger.dev.storj.io/trace/72f83fe3826be26a FAIL (2025-12-09 01:12:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 81.171.14.70:4026 , piece_id=PMFM6375JSHDHIOMRYSDP6JTVLU4H2MEMVJ2OMNS7EWETSZGNBDQ, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.736982501 (81h42m22.434550817s) [noise=true , address=81.171.14.70:4028 , piece_id=T5FWRHF3Q3P5ZJTJG6M4LSE5I3NMJZNEXZJAJ5L6QO3INDLYPKLQ, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.737310294 (81h42m22.43487861s) [noise=true , address=185.167.204.206:4048 , piece_id=AFESBI4TIYTKVGYGF5D237OCNFMWHDSJJQPYMODHPXDFGCY2M5QA, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.737645891 (81h42m22.435214207s) [noise=true , address=81.171.14.70:4025 , piece_id=UY5AW5OLUAXASGTK5QP26MHVXNEFLD43IWNNASCVPQTUT6FLVERQ, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.737933959 (81h42m22.435502275s) [noise=true , address=89.149.201.147:4011 , piece_id=U4BZXR7VS7ZRGZUDZ4RBROZXYLLOPURMWVGY3A6CP4EZEODG7IHA, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.738193974 (81h42m22.43576228s) [noise=true , address=89.149.201.147:4015 , piece_id=VTR7LPHSIZUGJIJVFRM2UKOKFPAKESYEK3O4Z3OKQM5QRPSMW5YA, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.738461063 (81h42m22.436029369s) [noise=true , address=81.171.14.70:4036 , piece_id=3JO6NP47U5QTOELQVBHRQFQV7XH7Q5TEJPENMPSMZAQWUQCZFGSA, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.740631258 (81h42m22.438199563s) [noise=true , address=185.167.204.206:4051 , piece_id=6OJULYV5IENLONFLA532YIQ73DDH6B57IPH2BDGS6G2QXONFAYUQ, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.740891504 (81h42m22.438459809s) [noise=true , address=81.171.14.70:4037 , piece_id=WFGPO7ES4IOPJYZITTG5M2GKCKDU7MWDBXRTGPJ4A6XLIO4F5WSQ, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.741179131 (81h42m22.438747447s) [noise=true , address=81.171.14.70:4039 , piece_id=5W77L52QLRHEXORVKN2ZUC4SGLBB4INC6G6NIVP2DG7D44HVBXIA, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.741476256 (81h42m22.439044572s) [noise=true , address=185.167.204.206:4067 , piece_id=PI4URUVBI6677BH4YYIVHHRSAAEZOXFXRA2MGQQYJENNZQNZZWEA, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:35.228809029 (81h42m22.926377345s) [upload=71700]: Next returned nil segment Dec 9 01:11:35.228817464 (81h42m22.926385781s) [upload=71700]: Waiting for error group managing segments... Dec 9 01:11:35.440098594 (81h42m23.13766691s) [noise=true , address=89.149.201.147:4011 , piece_id=U4BZXR7VS7ZRGZUDZ4RBROZXYLLOPURMWVGY3A6CP4EZEODG7IHA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.454769522 (81h42m23.152337828s) [noise=true , address=81.171.14.70:4025 , piece_id=UY5AW5OLUAXASGTK5QP26MHVXNEFLD43IWNNASCVPQTUT6FLVERQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.465162708 (81h42m23.162731014s) [noise=true , address=81.171.14.70:4034 , piece_id=3P7XHU7ZBRY5LVHWOXJ3BXXIS7EK3AU6SXY77FDKVMLS3S7EZR3A, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.469372954 (81h42m23.16694126s) [noise=true , address=185.167.204.206:4051 , piece_id=6OJULYV5IENLONFLA532YIQ73DDH6B57IPH2BDGS6G2QXONFAYUQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.587738791 (81h42m23.285307107s) [noise=true , address=185.167.204.206:4060 , piece_id=5PIIM7UXFOG4YKK2HQB4EHU3VEWR5UY37DPV4U652VFJPVO6HFWA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.740052040 (81h42m23.437620357s) [noise=true , address=89.149.201.147:4018 , piece_id=DO2SPAA5BVRVBUUJAZNZO5TYFEQ3R75FZCQLKTX6WCFSXBIKCGZQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.749418068 (81h42m23.446986384s) [noise=true , address=89.149.201.147:4002 , piece_id=6B64PAGFBVG7CLO2DFUJ42RFKCDYJFWSQRZANCZIRHH2HLZPNM2Q, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.753047188 (81h42m23.450615504s) [noise=true , address=89.149.201.147:4010 , piece_id=I43ACOF6RY52TSJRZOUBCJHKMUUOX52IR6MCSK66E7W52OQATNJQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.766977372 (81h42m23.464545677s) [noise=true , address=185.167.204.206:4071 , piece_id=OMTBSORG7SUYBUBJ6FDN3HOWT5RWRDW2IKAMPAMVM2P4QBGVY3QQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.921179428 (81h42m23.618747744s) [noise=true , address=81.171.14.70:4026 , piece_id=PMFM6375JSHDHIOMRYSDP6JTVLU4H2MEMVJ2OMNS7EWETSZGNBDQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.927181501 (81h42m23.624749817s) [noise=true , address=81.171.14.70:4035 , piece_id=MWIBKDF6CVXNDAOW7NVCI3FMO74TRUVD7Q3IRM5P2KVZGAXP6EMA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.933498451 (81h42m23.631066758s) [noise=true , address=81.171.14.70:4043 , piece_id=QVHYA7LKAK5IU7YUE24V2MROVELXVBC3NQ3PPO6XQ5PSUTGNW6GA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.003002376 (81h42m23.700570713s) [noise=true , address=185.167.204.206:4066 , piece_id=6OW3QB3X27IRPEBSCLJC2HZZBPF5UKOJNJNXUN47CEDKXGULV5HA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.006937678 (81h42m23.704505985s) [noise=true , address=81.171.14.70:4028 , piece_id=T5FWRHF3Q3P5ZJTJG6M4LSE5I3NMJZNEXZJAJ5L6QO3INDLYPKLQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.053939967 (81h42m23.751508273s) [noise=true , address=89.149.201.147:4015 , piece_id=VTR7LPHSIZUGJIJVFRM2UKOKFPAKESYEK3O4Z3OKQM5QRPSMW5YA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.088360861 (81h42m23.785929168s) [noise=true , address=185.167.204.206:4037 , piece_id=MVVUHY7CUU4NPULH6HDO4RKQ6F577VUIDHL5UM5KIB3ZDGNVP7JQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.131953010 (81h42m23.829521326s) [noise=true , address=185.167.204.206:4065 , piece_id=4UHGLV764LZISE23FWVH4FH77WTER2CV2AJJT7M5CM7BRWVEQ5XA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.414993819 (81h42m24.112562135s) [noise=true , address=81.171.14.70:4037 , piece_id=WFGPO7ES4IOPJYZITTG5M2GKCKDU7MWDBXRTGPJ4A6XLIO4F5WSQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.464634196 (81h42m24.162202512s) [noise=true , address=81.171.14.70:4039 , piece_id=5W77L52QLRHEXORVKN2ZUC4SGLBB4INC6G6NIVP2DG7D44HVBXIA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.497685902 (81h42m24.195254218s) [noise=true , address=89.149.201.147:4004 , piece_id=TGWFLCMQI6WAQZMS47DB2AQGO7MPVGXKZKXRRGMPRDNAHDFWWHNA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.598648698 (81h42m24.296217014s) [noise=true , address=89.149.201.147:4001 , piece_id=MDLJVVD75LKZQ5MKZSSXCVTJQMNQVMKSYKD5ZSHADK76W32IZGSQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.639249747 (81h42m24.336818063s) [noise=true , address=89.149.201.147:4024 , piece_id=IGEVVIYG2ZF4USOCDBANXBMJ5FVP7COL2DPCOOAVUR7WOJKWZXPA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.941575012 (81h42m24.639143328s) [noise=true , address=185.167.204.206:4067 , piece_id=PI4URUVBI6677BH4YYIVHHRSAAEZOXFXRA2MGQQYJENNZQNZZWEA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:37.065891045 (81h42m24.76345935s) [noise=true , address=185.167.204.206:4048 , piece_id=AFESBI4TIYTKVGYGF5D237OCNFMWHDSJJQPYMODHPXDFGCY2M5QA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:37.224380189 (81h42m24.921948495s) [noise=true , address=185.167.204.205:4001 , piece_id=I2QSUBMTV6W3QSRC4442KKR6KQODRRO7VU7JKCSKYARWODSRGIYQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:37.441711723 (81h42m25.139280029s) [noise=true , address=185.167.204.205:4023 , piece_id=XWBXYF5YY46SWJCXT6YL4CR54M72YDTAZLQX3NIFZCJUCCHRPOEQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:37.441725298 (81h42m25.139293604s) [seg_pos={0 0}, upload=71700]: Segment reached optimal threshold of 26 pieces. Dec 9 01:11:37.441876681 (81h42m25.139444997s) [noise=true , address=185.167.204.206:4045 , piece_id=YYEWRK3YRMJ7Z3UNZW6HVB7IMVLPWJZCD5W77KEOQQ577KXJKWAA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Dec 9 01:11:37.441883634 (81h42m25.13945195s) [noise=true , address=185.167.204.206:4045 , piece_id=YYEWRK3YRMJ7Z3UNZW6HVB7IMVLPWJZCD5W77KEOQQ577KXJKWAA, seg_pos={0 0}, upload=71700]: Long tail cancellation due to optimal threshold Dec 9 01:11:37.441984292 (81h42m25.139552598s) [noise=true , address=81.171.14.70:4036 , piece_id=3JO6NP47U5QTOELQVBHRQFQV7XH7Q5TEJPENMPSMZAQWUQCZFGSA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TcRxUCpNQVo3KLtKPK1zc6NXR4AdRCvqH4gtxticRH5MrKhVq): protocol: expected piece hash; context canceled; EOF Dec 9 01:11:37.441989592 (81h42m25.139557898s) [noise=true , address=81.171.14.70:4036 , piece_id=3JO6NP47U5QTOELQVBHRQFQV7XH7Q5TEJPENMPSMZAQWUQCZFGSA, seg_pos={0 0}, upload=71700]: Long tail cancellation due to optimal threshold Dec 9 01:11:37.442002566 (81h42m25.139570882s) [seg_pos={0 0}, upload=71700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:11:37.442054704 (81h42m25.13962301s) [upload=71700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:11:37.442058030 (81h42m25.139626346s) [upload=71700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:11:37.582834244 (81h42m25.28040256s) [upload=71700]: Done uploading segments... Dec 9 01:11:37.582848541 (81h42m25.280416857s) [upload=71700]: Upload finished. err: <nil> Duration: 4.330369059s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 14 out of 60 is failed (threshold=3) [...X........X...XX.....XX..X...X.....X.X.......X...X.....X.X] duration: 23.693510817s trace: https://jaeger.dev.storj.io/trace/72f83fe3826be26a FAIL (2025-12-09 01:11:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 81.171.14.70:4026 , piece_id=PMFM6375JSHDHIOMRYSDP6JTVLU4H2MEMVJ2OMNS7EWETSZGNBDQ, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.736982501 (81h42m22.434550817s) [noise=true , address=81.171.14.70:4028 , piece_id=T5FWRHF3Q3P5ZJTJG6M4LSE5I3NMJZNEXZJAJ5L6QO3INDLYPKLQ, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.737310294 (81h42m22.43487861s) [noise=true , address=185.167.204.206:4048 , piece_id=AFESBI4TIYTKVGYGF5D237OCNFMWHDSJJQPYMODHPXDFGCY2M5QA, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.737645891 (81h42m22.435214207s) [noise=true , address=81.171.14.70:4025 , piece_id=UY5AW5OLUAXASGTK5QP26MHVXNEFLD43IWNNASCVPQTUT6FLVERQ, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.737933959 (81h42m22.435502275s) [noise=true , address=89.149.201.147:4011 , piece_id=U4BZXR7VS7ZRGZUDZ4RBROZXYLLOPURMWVGY3A6CP4EZEODG7IHA, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.738193974 (81h42m22.43576228s) [noise=true , address=89.149.201.147:4015 , piece_id=VTR7LPHSIZUGJIJVFRM2UKOKFPAKESYEK3O4Z3OKQM5QRPSMW5YA, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.738461063 (81h42m22.436029369s) [noise=true , address=81.171.14.70:4036 , piece_id=3JO6NP47U5QTOELQVBHRQFQV7XH7Q5TEJPENMPSMZAQWUQCZFGSA, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.740631258 (81h42m22.438199563s) [noise=true , address=185.167.204.206:4051 , piece_id=6OJULYV5IENLONFLA532YIQ73DDH6B57IPH2BDGS6G2QXONFAYUQ, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.740891504 (81h42m22.438459809s) [noise=true , address=81.171.14.70:4037 , piece_id=WFGPO7ES4IOPJYZITTG5M2GKCKDU7MWDBXRTGPJ4A6XLIO4F5WSQ, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.741179131 (81h42m22.438747447s) [noise=true , address=81.171.14.70:4039 , piece_id=5W77L52QLRHEXORVKN2ZUC4SGLBB4INC6G6NIVP2DG7D44HVBXIA, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:34.741476256 (81h42m22.439044572s) [noise=true , address=185.167.204.206:4067 , piece_id=PI4URUVBI6677BH4YYIVHHRSAAEZOXFXRA2MGQQYJENNZQNZZWEA, seg_pos={0 0}, upload=71700]: Uploading piece... Dec 9 01:11:35.228809029 (81h42m22.926377345s) [upload=71700]: Next returned nil segment Dec 9 01:11:35.228817464 (81h42m22.926385781s) [upload=71700]: Waiting for error group managing segments... Dec 9 01:11:35.440098594 (81h42m23.13766691s) [noise=true , address=89.149.201.147:4011 , piece_id=U4BZXR7VS7ZRGZUDZ4RBROZXYLLOPURMWVGY3A6CP4EZEODG7IHA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.454769522 (81h42m23.152337828s) [noise=true , address=81.171.14.70:4025 , piece_id=UY5AW5OLUAXASGTK5QP26MHVXNEFLD43IWNNASCVPQTUT6FLVERQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.465162708 (81h42m23.162731014s) [noise=true , address=81.171.14.70:4034 , piece_id=3P7XHU7ZBRY5LVHWOXJ3BXXIS7EK3AU6SXY77FDKVMLS3S7EZR3A, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.469372954 (81h42m23.16694126s) [noise=true , address=185.167.204.206:4051 , piece_id=6OJULYV5IENLONFLA532YIQ73DDH6B57IPH2BDGS6G2QXONFAYUQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.587738791 (81h42m23.285307107s) [noise=true , address=185.167.204.206:4060 , piece_id=5PIIM7UXFOG4YKK2HQB4EHU3VEWR5UY37DPV4U652VFJPVO6HFWA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.740052040 (81h42m23.437620357s) [noise=true , address=89.149.201.147:4018 , piece_id=DO2SPAA5BVRVBUUJAZNZO5TYFEQ3R75FZCQLKTX6WCFSXBIKCGZQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.749418068 (81h42m23.446986384s) [noise=true , address=89.149.201.147:4002 , piece_id=6B64PAGFBVG7CLO2DFUJ42RFKCDYJFWSQRZANCZIRHH2HLZPNM2Q, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.753047188 (81h42m23.450615504s) [noise=true , address=89.149.201.147:4010 , piece_id=I43ACOF6RY52TSJRZOUBCJHKMUUOX52IR6MCSK66E7W52OQATNJQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.766977372 (81h42m23.464545677s) [noise=true , address=185.167.204.206:4071 , piece_id=OMTBSORG7SUYBUBJ6FDN3HOWT5RWRDW2IKAMPAMVM2P4QBGVY3QQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.921179428 (81h42m23.618747744s) [noise=true , address=81.171.14.70:4026 , piece_id=PMFM6375JSHDHIOMRYSDP6JTVLU4H2MEMVJ2OMNS7EWETSZGNBDQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.927181501 (81h42m23.624749817s) [noise=true , address=81.171.14.70:4035 , piece_id=MWIBKDF6CVXNDAOW7NVCI3FMO74TRUVD7Q3IRM5P2KVZGAXP6EMA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:35.933498451 (81h42m23.631066758s) [noise=true , address=81.171.14.70:4043 , piece_id=QVHYA7LKAK5IU7YUE24V2MROVELXVBC3NQ3PPO6XQ5PSUTGNW6GA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.003002376 (81h42m23.700570713s) [noise=true , address=185.167.204.206:4066 , piece_id=6OW3QB3X27IRPEBSCLJC2HZZBPF5UKOJNJNXUN47CEDKXGULV5HA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.006937678 (81h42m23.704505985s) [noise=true , address=81.171.14.70:4028 , piece_id=T5FWRHF3Q3P5ZJTJG6M4LSE5I3NMJZNEXZJAJ5L6QO3INDLYPKLQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.053939967 (81h42m23.751508273s) [noise=true , address=89.149.201.147:4015 , piece_id=VTR7LPHSIZUGJIJVFRM2UKOKFPAKESYEK3O4Z3OKQM5QRPSMW5YA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.088360861 (81h42m23.785929168s) [noise=true , address=185.167.204.206:4037 , piece_id=MVVUHY7CUU4NPULH6HDO4RKQ6F577VUIDHL5UM5KIB3ZDGNVP7JQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.131953010 (81h42m23.829521326s) [noise=true , address=185.167.204.206:4065 , piece_id=4UHGLV764LZISE23FWVH4FH77WTER2CV2AJJT7M5CM7BRWVEQ5XA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.414993819 (81h42m24.112562135s) [noise=true , address=81.171.14.70:4037 , piece_id=WFGPO7ES4IOPJYZITTG5M2GKCKDU7MWDBXRTGPJ4A6XLIO4F5WSQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.464634196 (81h42m24.162202512s) [noise=true , address=81.171.14.70:4039 , piece_id=5W77L52QLRHEXORVKN2ZUC4SGLBB4INC6G6NIVP2DG7D44HVBXIA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.497685902 (81h42m24.195254218s) [noise=true , address=89.149.201.147:4004 , piece_id=TGWFLCMQI6WAQZMS47DB2AQGO7MPVGXKZKXRRGMPRDNAHDFWWHNA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.598648698 (81h42m24.296217014s) [noise=true , address=89.149.201.147:4001 , piece_id=MDLJVVD75LKZQ5MKZSSXCVTJQMNQVMKSYKD5ZSHADK76W32IZGSQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.639249747 (81h42m24.336818063s) [noise=true , address=89.149.201.147:4024 , piece_id=IGEVVIYG2ZF4USOCDBANXBMJ5FVP7COL2DPCOOAVUR7WOJKWZXPA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:36.941575012 (81h42m24.639143328s) [noise=true , address=185.167.204.206:4067 , piece_id=PI4URUVBI6677BH4YYIVHHRSAAEZOXFXRA2MGQQYJENNZQNZZWEA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:37.065891045 (81h42m24.76345935s) [noise=true , address=185.167.204.206:4048 , piece_id=AFESBI4TIYTKVGYGF5D237OCNFMWHDSJJQPYMODHPXDFGCY2M5QA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:37.224380189 (81h42m24.921948495s) [noise=true , address=185.167.204.205:4001 , piece_id=I2QSUBMTV6W3QSRC4442KKR6KQODRRO7VU7JKCSKYARWODSRGIYQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:37.441711723 (81h42m25.139280029s) [noise=true , address=185.167.204.205:4023 , piece_id=XWBXYF5YY46SWJCXT6YL4CR54M72YDTAZLQX3NIFZCJUCCHRPOEQ, seg_pos={0 0}, upload=71700]: Done uploading piece. err: <nil> Dec 9 01:11:37.441725298 (81h42m25.139293604s) [seg_pos={0 0}, upload=71700]: Segment reached optimal threshold of 26 pieces. Dec 9 01:11:37.441876681 (81h42m25.139444997s) [noise=true , address=185.167.204.206:4045 , piece_id=YYEWRK3YRMJ7Z3UNZW6HVB7IMVLPWJZCD5W77KEOQQ577KXJKWAA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq): protocol: expected piece hash; context canceled; EOF Dec 9 01:11:37.441883634 (81h42m25.13945195s) [noise=true , address=185.167.204.206:4045 , piece_id=YYEWRK3YRMJ7Z3UNZW6HVB7IMVLPWJZCD5W77KEOQQ577KXJKWAA, seg_pos={0 0}, upload=71700]: Long tail cancellation due to optimal threshold Dec 9 01:11:37.441984292 (81h42m25.139552598s) [noise=true , address=81.171.14.70:4036 , piece_id=3JO6NP47U5QTOELQVBHRQFQV7XH7Q5TEJPENMPSMZAQWUQCZFGSA, seg_pos={0 0}, upload=71700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TcRxUCpNQVo3KLtKPK1zc6NXR4AdRCvqH4gtxticRH5MrKhVq): protocol: expected piece hash; context canceled; EOF Dec 9 01:11:37.441989592 (81h42m25.139557898s) [noise=true , address=81.171.14.70:4036 , piece_id=3JO6NP47U5QTOELQVBHRQFQV7XH7Q5TEJPENMPSMZAQWUQCZFGSA, seg_pos={0 0}, upload=71700]: Long tail cancellation due to optimal threshold Dec 9 01:11:37.442002566 (81h42m25.139570882s) [seg_pos={0 0}, upload=71700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:11:37.442054704 (81h42m25.13962301s) [upload=71700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:11:37.442058030 (81h42m25.139626346s) [upload=71700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:11:37.582834244 (81h42m25.28040256s) [upload=71700]: Done uploading segments... Dec 9 01:11:37.582848541 (81h42m25.280416857s) [upload=71700]: Upload finished. err: <nil> Duration: 23.693545339s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 407.630252ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:11:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.643366ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 13 out of 60 is failed (threshold=3) [....X........X...XX.....XX..X...X.....X.X.......X...X.....X.] duration: 30.963991791s trace: https://jaeger.dev.storj.io/trace/678aaddd1a941b20 FAIL (2025-12-09 01:10:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== , address=185.167.204.205:4023 , piece_id=Y42URT6OQIYVAQEV4YPAWGZASTODN5US67IDBNKS4QDRC344QJ7A, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.412630360 (81h40m9.110198676s) [noise=true , address=89.149.201.147:4007 , piece_id=UFZA7QSH6DSCR4OW4FQC2MQPXDULL32PLJG5RSZVULA2GQKUAKOA, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.412764139 (81h40m9.110332456s) [noise=true , address=81.171.14.70:4045 , piece_id=4U4D2KPETMRUIX72K7BOP4QOQV2QBU53Y5VSVQ3VIDP7SCJZ6ZSQ, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.412999830 (81h40m9.110568146s) [noise=true , address=89.149.201.147:4019 , piece_id=OJQIU6HJ74WDLD44VO66UT5LH3FUATW6RFFE6TFQSHXEL433YSMQ, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.413035486 (81h40m9.110603802s) [noise=true , address=81.171.14.70:4033 , piece_id=RZ65JQZ6ZEBCOPMHC55C7G3FRNW5W7S67VIDGDZ6I2SNK4EVVK4Q, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.413353400 (81h40m9.110921707s) [noise=true , address=185.167.204.205:4017 , piece_id=CIIWYVOJNEGTLYX2VYBDO67HG55RV6K4OJCLKDMFDRRHZFYQRGFA, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.413431877 (81h40m9.111000193s) [noise=true , address=89.149.201.147:4024 , piece_id=VLHOYFKQ75MUXRCET27NGPC3KQGRRPVDCUJLGKO3O5JJSPNJCWCQ, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.413720205 (81h40m9.111288521s) [noise=true , address=81.171.14.70:4042 , piece_id=FHYP4EBPX2JBKMBCNO5XKDLIU5PXA4E2YPIVR3KRXFLL7Y5JON7Q, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.413983978 (81h40m9.111552294s) [noise=true , address=81.171.14.70:4031 , piece_id=NEWDJUQAB7H3R45R6GVJTCETN2HTIKRHVNKMGTM5A5EO3Y7SXNWA, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.414555787 (81h40m9.112124104s) [noise=true , address=89.149.201.147:4022 , piece_id=EQPUNQGOQ65MTN4K2ONZQSJXDNG7FADCUFAWLSDK6CGNDYMJ3F3A, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.414825682 (81h40m9.112393987s) [noise=true , address=185.167.204.205:4036 , piece_id=46KMAYL5Y43G2SHCXCIG2ORFWAGJAZDHYH477Q56HUMNOFVJ6D3A, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.910023977 (81h40m9.607592283s) [upload=71671]: Next returned nil segment Dec 9 01:09:21.910032223 (81h40m9.607600529s) [upload=71671]: Waiting for error group managing segments... Dec 9 01:09:22.112053433 (81h40m9.809621739s) [noise=true , address=185.167.204.205:4017 , piece_id=CIIWYVOJNEGTLYX2VYBDO67HG55RV6K4OJCLKDMFDRRHZFYQRGFA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.112149503 (81h40m9.809717809s) [noise=true , address=185.167.204.205:4023 , piece_id=Y42URT6OQIYVAQEV4YPAWGZASTODN5US67IDBNKS4QDRC344QJ7A, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.113687626 (81h40m9.811255932s) [noise=true , address=185.167.204.205:4001 , piece_id=2TRYX6OK3XZKKCVVHN6SR6ZX5XJTJI4W7UGXRRPX5FUKNWZCVUAA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.167457357 (81h40m9.865025684s) [noise=true , address=89.149.201.147:4024 , piece_id=VLHOYFKQ75MUXRCET27NGPC3KQGRRPVDCUJLGKO3O5JJSPNJCWCQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.171867326 (81h40m9.869435661s) [noise=true , address=185.167.204.206:4062 , piece_id=SO4WCQO2Z34ACXCSHMLPVEUN3EGXWJFSLNC5Q5K4NGQBFEBK25FQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.201789815 (81h40m9.899358122s) [noise=true , address=185.167.204.206:4048 , piece_id=3SMPHTTBPU5IDQ5M4SPFLVSWB4BEGWCZC4UEZ7BX53YG7HBZ5FEQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.452651008 (81h40m10.150219324s) [noise=true , address=81.171.14.70:4025 , piece_id=AWEQYSMLCHFL3TIY2VD677M74YB4R3IX4S7KNLCEFRSLGJ5BXH7A, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.539815847 (81h40m10.237384194s) [noise=true , address=81.171.14.70:4031 , piece_id=NEWDJUQAB7H3R45R6GVJTCETN2HTIKRHVNKMGTM5A5EO3Y7SXNWA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.611551390 (81h40m10.309119727s) [noise=true , address=185.167.204.205:4030 , piece_id=5EUNXCYK3HYKEJQGBBQKSN4GZKQQNGF3LOOVVUISG54ASPKRW4AQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.731734684 (81h40m10.42930299s) [noise=true , address=89.149.201.147:4007 , piece_id=UFZA7QSH6DSCR4OW4FQC2MQPXDULL32PLJG5RSZVULA2GQKUAKOA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.754973187 (81h40m10.452541533s) [noise=true , address=89.149.201.147:4018 , piece_id=3A3WSE2ASKVFAJH4Y2XVCKVJDSZY3YF5IXKFCPS2NFGXYN6I5FGQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.774336671 (81h40m10.471905008s) [noise=true , address=81.171.14.70:4035 , piece_id=OGIEU5EVWXNB55ICBLUT6P72VCHE6C37TWW57ZCPCTUC6MBPHARA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.795521479 (81h40m10.493089815s) [noise=true , address=81.171.14.70:4027 , piece_id=44EAB2E6OGC2B5DPEK2ECYH436MZYXPRHP5BIE5WVQO6Z5FLUUQQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.829662389 (81h40m10.527230706s) [noise=true , address=89.149.201.147:4022 , piece_id=EQPUNQGOQ65MTN4K2ONZQSJXDNG7FADCUFAWLSDK6CGNDYMJ3F3A, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.895003348 (81h40m10.592571664s) [noise=true , address=81.171.14.70:4040 , piece_id=FVSSDKKAFBKMOUU4JBP4SZIMCD4F5ZCLOMVYU7HF7YGY5DKU4MXA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.960202200 (81h40m10.657770506s) [noise=true , address=185.167.204.205:4031 , piece_id=G7XE2LQRL4NFHABLQ2HRCPDGMGYLP22CPSXZGHY67PUSDW4VUITA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.007106645 (81h40m10.704674951s) [noise=true , address=89.149.201.147:4019 , piece_id=OJQIU6HJ74WDLD44VO66UT5LH3FUATW6RFFE6TFQSHXEL433YSMQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.121979979 (81h40m10.819548315s) [noise=true , address=81.171.14.70:4045 , piece_id=4U4D2KPETMRUIX72K7BOP4QOQV2QBU53Y5VSVQ3VIDP7SCJZ6ZSQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.337445955 (81h40m11.035014271s) [noise=true , address=81.171.14.70:4042 , piece_id=FHYP4EBPX2JBKMBCNO5XKDLIU5PXA4E2YPIVR3KRXFLL7Y5JON7Q, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.637434765 (81h40m11.335003072s) [noise=true , address=81.171.14.70:4029 , piece_id=YKQWBVMTH7CRLECLQNKEHYKOWAWM5LFLTFD7QF4MURIZADR76N6Q, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.750553671 (81h40m11.448121987s) [noise=true , address=81.171.14.70:4039 , piece_id=WUPL7WOOV3374GZX745T4LWSZDYQPJFE6JBM7GHZAB45BJWFZTVQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.803389858 (81h40m11.500958174s) [noise=true , address=185.167.204.205:4036 , piece_id=46KMAYL5Y43G2SHCXCIG2ORFWAGJAZDHYH477Q56HUMNOFVJ6D3A, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.861628418 (81h40m11.559196753s) [noise=true , address=89.149.201.147:4017 , piece_id=M4OSJ6KGRV6DXX7B6JK5D4FNE3UMV7SL6G35OX6VSDVFBY3NSQ7A, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.886341696 (81h40m11.583910002s) [noise=true , address=81.171.14.70:4044 , piece_id=YFK2RWGQWDCIHR7FQIMNRLU7GOU2U3V7DS2K6C422ZSVWKORYLXA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:24.398170210 (81h40m12.095738516s) [noise=true , address=89.149.201.147:4008 , piece_id=FD5MWAMHARCAZ5CWQZXGMTWL6GFIU4GR6QFPFYJRYMRZUZX7VATA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:24.556742259 (81h40m12.254310595s) [noise=true , address=81.171.14.70:4033 , piece_id=RZ65JQZ6ZEBCOPMHC55C7G3FRNW5W7S67VIDGDZ6I2SNK4EVVK4Q, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:24.556762888 (81h40m12.254331234s) [seg_pos={0 0}, upload=71671]: Segment reached optimal threshold of 26 pieces. Dec 9 01:09:24.556998968 (81h40m12.254567304s) [noise=true , address=89.149.201.147:4016 , piece_id=T4HDCRNSN5TLHHA2O7W4637IUKDZRFPZTPXUMXHGA65EIY5KI5VQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 9 01:09:24.557009167 (81h40m12.254577513s) [noise=true , address=89.149.201.147:4016 , piece_id=T4HDCRNSN5TLHHA2O7W4637IUKDZRFPZTPXUMXHGA65EIY5KI5VQ, seg_pos={0 0}, upload=71671]: Long tail cancellation due to optimal threshold Dec 9 01:09:24.557024286 (81h40m12.254592622s) [noise=true , address=185.167.204.205:4026 , piece_id=ZKCT2IFB6WGHU4SQ22AZFJDCHDISQRWLX5PKJ3WGJXPRU27FKP4A, seg_pos={0 0}, upload=71671]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4): protocol: expected piece hash; context canceled; EOF Dec 9 01:09:24.557037059 (81h40m12.254605396s) [noise=true , address=185.167.204.205:4026 , piece_id=ZKCT2IFB6WGHU4SQ22AZFJDCHDISQRWLX5PKJ3WGJXPRU27FKP4A, seg_pos={0 0}, upload=71671]: Long tail cancellation due to optimal threshold Dec 9 01:09:24.557061796 (81h40m12.254630132s) [seg_pos={0 0}, upload=71671]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:09:24.557142536 (81h40m12.254710873s) [upload=71671]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:09:24.557148177 (81h40m12.254716513s) [upload=71671]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:09:24.700069199 (81h40m12.397637544s) [upload=71671]: Done uploading segments... Dec 9 01:09:24.700086420 (81h40m12.397654757s) [upload=71671]: Upload finished. err: <nil> Duration: 30.964014494s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 426.683841ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:10:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 426.709008ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 13 out of 60 is failed (threshold=3) [.....X........X...XX.....XX..X...X.....X.X.......X...X.....X] duration: 18.111074115s trace: https://jaeger.dev.storj.io/trace/678aaddd1a941b20 FAIL (2025-12-09 01:09:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== , address=185.167.204.205:4023 , piece_id=Y42URT6OQIYVAQEV4YPAWGZASTODN5US67IDBNKS4QDRC344QJ7A, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.412630360 (81h40m9.110198676s) [noise=true , address=89.149.201.147:4007 , piece_id=UFZA7QSH6DSCR4OW4FQC2MQPXDULL32PLJG5RSZVULA2GQKUAKOA, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.412764139 (81h40m9.110332456s) [noise=true , address=81.171.14.70:4045 , piece_id=4U4D2KPETMRUIX72K7BOP4QOQV2QBU53Y5VSVQ3VIDP7SCJZ6ZSQ, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.412999830 (81h40m9.110568146s) [noise=true , address=89.149.201.147:4019 , piece_id=OJQIU6HJ74WDLD44VO66UT5LH3FUATW6RFFE6TFQSHXEL433YSMQ, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.413035486 (81h40m9.110603802s) [noise=true , address=81.171.14.70:4033 , piece_id=RZ65JQZ6ZEBCOPMHC55C7G3FRNW5W7S67VIDGDZ6I2SNK4EVVK4Q, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.413353400 (81h40m9.110921707s) [noise=true , address=185.167.204.205:4017 , piece_id=CIIWYVOJNEGTLYX2VYBDO67HG55RV6K4OJCLKDMFDRRHZFYQRGFA, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.413431877 (81h40m9.111000193s) [noise=true , address=89.149.201.147:4024 , piece_id=VLHOYFKQ75MUXRCET27NGPC3KQGRRPVDCUJLGKO3O5JJSPNJCWCQ, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.413720205 (81h40m9.111288521s) [noise=true , address=81.171.14.70:4042 , piece_id=FHYP4EBPX2JBKMBCNO5XKDLIU5PXA4E2YPIVR3KRXFLL7Y5JON7Q, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.413983978 (81h40m9.111552294s) [noise=true , address=81.171.14.70:4031 , piece_id=NEWDJUQAB7H3R45R6GVJTCETN2HTIKRHVNKMGTM5A5EO3Y7SXNWA, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.414555787 (81h40m9.112124104s) [noise=true , address=89.149.201.147:4022 , piece_id=EQPUNQGOQ65MTN4K2ONZQSJXDNG7FADCUFAWLSDK6CGNDYMJ3F3A, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.414825682 (81h40m9.112393987s) [noise=true , address=185.167.204.205:4036 , piece_id=46KMAYL5Y43G2SHCXCIG2ORFWAGJAZDHYH477Q56HUMNOFVJ6D3A, seg_pos={0 0}, upload=71671]: Uploading piece... Dec 9 01:09:21.910023977 (81h40m9.607592283s) [upload=71671]: Next returned nil segment Dec 9 01:09:21.910032223 (81h40m9.607600529s) [upload=71671]: Waiting for error group managing segments... Dec 9 01:09:22.112053433 (81h40m9.809621739s) [noise=true , address=185.167.204.205:4017 , piece_id=CIIWYVOJNEGTLYX2VYBDO67HG55RV6K4OJCLKDMFDRRHZFYQRGFA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.112149503 (81h40m9.809717809s) [noise=true , address=185.167.204.205:4023 , piece_id=Y42URT6OQIYVAQEV4YPAWGZASTODN5US67IDBNKS4QDRC344QJ7A, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.113687626 (81h40m9.811255932s) [noise=true , address=185.167.204.205:4001 , piece_id=2TRYX6OK3XZKKCVVHN6SR6ZX5XJTJI4W7UGXRRPX5FUKNWZCVUAA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.167457357 (81h40m9.865025684s) [noise=true , address=89.149.201.147:4024 , piece_id=VLHOYFKQ75MUXRCET27NGPC3KQGRRPVDCUJLGKO3O5JJSPNJCWCQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.171867326 (81h40m9.869435661s) [noise=true , address=185.167.204.206:4062 , piece_id=SO4WCQO2Z34ACXCSHMLPVEUN3EGXWJFSLNC5Q5K4NGQBFEBK25FQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.201789815 (81h40m9.899358122s) [noise=true , address=185.167.204.206:4048 , piece_id=3SMPHTTBPU5IDQ5M4SPFLVSWB4BEGWCZC4UEZ7BX53YG7HBZ5FEQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.452651008 (81h40m10.150219324s) [noise=true , address=81.171.14.70:4025 , piece_id=AWEQYSMLCHFL3TIY2VD677M74YB4R3IX4S7KNLCEFRSLGJ5BXH7A, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.539815847 (81h40m10.237384194s) [noise=true , address=81.171.14.70:4031 , piece_id=NEWDJUQAB7H3R45R6GVJTCETN2HTIKRHVNKMGTM5A5EO3Y7SXNWA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.611551390 (81h40m10.309119727s) [noise=true , address=185.167.204.205:4030 , piece_id=5EUNXCYK3HYKEJQGBBQKSN4GZKQQNGF3LOOVVUISG54ASPKRW4AQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.731734684 (81h40m10.42930299s) [noise=true , address=89.149.201.147:4007 , piece_id=UFZA7QSH6DSCR4OW4FQC2MQPXDULL32PLJG5RSZVULA2GQKUAKOA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.754973187 (81h40m10.452541533s) [noise=true , address=89.149.201.147:4018 , piece_id=3A3WSE2ASKVFAJH4Y2XVCKVJDSZY3YF5IXKFCPS2NFGXYN6I5FGQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.774336671 (81h40m10.471905008s) [noise=true , address=81.171.14.70:4035 , piece_id=OGIEU5EVWXNB55ICBLUT6P72VCHE6C37TWW57ZCPCTUC6MBPHARA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.795521479 (81h40m10.493089815s) [noise=true , address=81.171.14.70:4027 , piece_id=44EAB2E6OGC2B5DPEK2ECYH436MZYXPRHP5BIE5WVQO6Z5FLUUQQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.829662389 (81h40m10.527230706s) [noise=true , address=89.149.201.147:4022 , piece_id=EQPUNQGOQ65MTN4K2ONZQSJXDNG7FADCUFAWLSDK6CGNDYMJ3F3A, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.895003348 (81h40m10.592571664s) [noise=true , address=81.171.14.70:4040 , piece_id=FVSSDKKAFBKMOUU4JBP4SZIMCD4F5ZCLOMVYU7HF7YGY5DKU4MXA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:22.960202200 (81h40m10.657770506s) [noise=true , address=185.167.204.205:4031 , piece_id=G7XE2LQRL4NFHABLQ2HRCPDGMGYLP22CPSXZGHY67PUSDW4VUITA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.007106645 (81h40m10.704674951s) [noise=true , address=89.149.201.147:4019 , piece_id=OJQIU6HJ74WDLD44VO66UT5LH3FUATW6RFFE6TFQSHXEL433YSMQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.121979979 (81h40m10.819548315s) [noise=true , address=81.171.14.70:4045 , piece_id=4U4D2KPETMRUIX72K7BOP4QOQV2QBU53Y5VSVQ3VIDP7SCJZ6ZSQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.337445955 (81h40m11.035014271s) [noise=true , address=81.171.14.70:4042 , piece_id=FHYP4EBPX2JBKMBCNO5XKDLIU5PXA4E2YPIVR3KRXFLL7Y5JON7Q, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.637434765 (81h40m11.335003072s) [noise=true , address=81.171.14.70:4029 , piece_id=YKQWBVMTH7CRLECLQNKEHYKOWAWM5LFLTFD7QF4MURIZADR76N6Q, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.750553671 (81h40m11.448121987s) [noise=true , address=81.171.14.70:4039 , piece_id=WUPL7WOOV3374GZX745T4LWSZDYQPJFE6JBM7GHZAB45BJWFZTVQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.803389858 (81h40m11.500958174s) [noise=true , address=185.167.204.205:4036 , piece_id=46KMAYL5Y43G2SHCXCIG2ORFWAGJAZDHYH477Q56HUMNOFVJ6D3A, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.861628418 (81h40m11.559196753s) [noise=true , address=89.149.201.147:4017 , piece_id=M4OSJ6KGRV6DXX7B6JK5D4FNE3UMV7SL6G35OX6VSDVFBY3NSQ7A, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:23.886341696 (81h40m11.583910002s) [noise=true , address=81.171.14.70:4044 , piece_id=YFK2RWGQWDCIHR7FQIMNRLU7GOU2U3V7DS2K6C422ZSVWKORYLXA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:24.398170210 (81h40m12.095738516s) [noise=true , address=89.149.201.147:4008 , piece_id=FD5MWAMHARCAZ5CWQZXGMTWL6GFIU4GR6QFPFYJRYMRZUZX7VATA, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:24.556742259 (81h40m12.254310595s) [noise=true , address=81.171.14.70:4033 , piece_id=RZ65JQZ6ZEBCOPMHC55C7G3FRNW5W7S67VIDGDZ6I2SNK4EVVK4Q, seg_pos={0 0}, upload=71671]: Done uploading piece. err: <nil> Dec 9 01:09:24.556762888 (81h40m12.254331234s) [seg_pos={0 0}, upload=71671]: Segment reached optimal threshold of 26 pieces. Dec 9 01:09:24.556998968 (81h40m12.254567304s) [noise=true , address=89.149.201.147:4016 , piece_id=T4HDCRNSN5TLHHA2O7W4637IUKDZRFPZTPXUMXHGA65EIY5KI5VQ, seg_pos={0 0}, upload=71671]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 9 01:09:24.557009167 (81h40m12.254577513s) [noise=true , address=89.149.201.147:4016 , piece_id=T4HDCRNSN5TLHHA2O7W4637IUKDZRFPZTPXUMXHGA65EIY5KI5VQ, seg_pos={0 0}, upload=71671]: Long tail cancellation due to optimal threshold Dec 9 01:09:24.557024286 (81h40m12.254592622s) [noise=true , address=185.167.204.205:4026 , piece_id=ZKCT2IFB6WGHU4SQ22AZFJDCHDISQRWLX5PKJ3WGJXPRU27FKP4A, seg_pos={0 0}, upload=71671]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4): protocol: expected piece hash; context canceled; EOF Dec 9 01:09:24.557037059 (81h40m12.254605396s) [noise=true , address=185.167.204.205:4026 , piece_id=ZKCT2IFB6WGHU4SQ22AZFJDCHDISQRWLX5PKJ3WGJXPRU27FKP4A, seg_pos={0 0}, upload=71671]: Long tail cancellation due to optimal threshold Dec 9 01:09:24.557061796 (81h40m12.254630132s) [seg_pos={0 0}, upload=71671]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:09:24.557142536 (81h40m12.254710873s) [upload=71671]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:09:24.557148177 (81h40m12.254716513s) [upload=71671]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:09:24.700069199 (81h40m12.397637544s) [upload=71671]: Done uploading segments... Dec 9 01:09:24.700086420 (81h40m12.397654757s) [upload=71671]: Upload finished. err: <nil> Duration: 18.111121334s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 12 out of 60 is failed (threshold=3) [......X........X...XX.....XX..X...X.....X.X.......X...X.....] duration: 7.571663551s trace: https://jaeger.dev.storj.io/trace/3f20fc47e98690b9 FAIL (2025-12-09 01:09:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== .167.204.206:4072 , piece_id=LNCOWESTFDPRT5GMUZWCT6K2GTVRL7SY7EPMCL3BEU7D2ET36CMA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.800352280 (81h36m24.497920596s) [noise=true , address=89.149.201.147:4006 , piece_id=DIG4CLXPEKHJ7NTU5FSXIQ7UF2AZRP3IS5VBQZDTCLYVNU7AGOQA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.801378487 (81h36m24.498946803s) [noise=true , address=81.171.14.70:4044 , piece_id=57QPDI374TVOD7IQAMV6O537BZTYKDJTGUMV6V4DT5HIO6TSXJZA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.801993136 (81h36m24.499561441s) [noise=true , address=185.167.204.206:4063 , piece_id=6EQHKKLC2WL23KZ6C64343Q7UVYZVCSDGG3IW72VI77IHHECC3NA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.803054798 (81h36m24.500623114s) [noise=true , address=81.171.14.70:4035 , piece_id=ZYJD6KKUQHEVQM753L3BOUVSWTRTGJ3QL4MPSRW32BYS7WI3NXCA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.804454994 (81h36m24.502023311s) [noise=true , address=89.149.201.147:4004 , piece_id=OSLAKDSTRU2HKYKMBVBR5PEXHPDX2JZ4YKJWFLZQHTLSYKBY6FGA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.806500335 (81h36m24.504068651s) [noise=true , address=81.171.14.70:4032 , piece_id=4YQ37JCLYWHN7NDROVCYAK7LJYAKG6LEFHJ3SUHLJ4XZK65GN77A, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.808007841 (81h36m24.505576157s) [noise=true , address=81.171.14.70:4039 , piece_id=UMVZ2DBSTQZI5RX2SBXZ54TOVJVGO7YL6LEKMQKCJ2PTFLXCWWAQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.810776653 (81h36m24.508344959s) [noise=true , address=89.149.201.147:4011 , piece_id=743ZJO643CQ6DWYJ7I5WWF3U35CEPHDISZVSK2NZKOUKJDOA27FA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.822630489 (81h36m24.520198825s) [noise=true , address=185.167.204.206:4051 , piece_id=ERNWXDJUTTJCTOLBJXN7GBF6ZHFLPA6NDQREYLXMKZSHKVYKVNGQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.824087431 (81h36m24.521655737s) [noise=true , address=185.167.204.206:4043 , piece_id=2KGX72ULTVLFVN75NLCCPGJWFGJGJCBHOG6GACIO7KAIO57HH6KQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:37.301186810 (81h36m24.998755126s) [upload=71621]: Next returned nil segment Dec 9 01:05:37.301200385 (81h36m24.998768691s) [upload=71621]: Waiting for error group managing segments... Dec 9 01:05:37.502191335 (81h36m25.199759652s) [noise=true , address=81.171.14.70:4032 , piece_id=4YQ37JCLYWHN7NDROVCYAK7LJYAKG6LEFHJ3SUHLJ4XZK65GN77A, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.524058687 (81h36m25.221627003s) [noise=true , address=81.171.14.70:4041 , piece_id=XKVCPTKZKNQ3GBHLHVPRSH6DXE6R4QGOSW34MYAI6FT7LCELQ5YQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.540043689 (81h36m25.237612005s) [noise=true , address=185.167.204.205:4013 , piece_id=TGYBDY5F4MYWOGX5E3TT2SF3WVQZFSYDYWXY4I6YA3IKK4LM6YXQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.559903370 (81h36m25.257471686s) [noise=true , address=89.149.201.147:4020 , piece_id=3V6MIKGBDVFHUEL7TC6JUW7P6IXTOR2JSFOYO7A3S2Q27QDWC5YQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.581565809 (81h36m25.279134125s) [noise=true , address=89.149.201.147:4004 , piece_id=OSLAKDSTRU2HKYKMBVBR5PEXHPDX2JZ4YKJWFLZQHTLSYKBY6FGA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.913526920 (81h36m25.611095256s) [noise=true , address=185.167.204.206:4051 , piece_id=ERNWXDJUTTJCTOLBJXN7GBF6ZHFLPA6NDQREYLXMKZSHKVYKVNGQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.076632645 (81h36m25.774200991s) [noise=true , address=81.171.14.70:4033 , piece_id=H65LLAS4JHUZGGMBIA5WC3XH4HYULEOQDY7HS7JKBIYFQFGSOFOQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.115906113 (81h36m25.813474449s) [noise=true , address=89.149.201.147:4018 , piece_id=A3YGMQFONONZSFRQTLZNIBFBNLJPL7JUQOSBR2WULNEE5UYFXTUQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.199498056 (81h36m25.897066392s) [noise=true , address=81.171.14.70:4037 , piece_id=PDBORKWMIIEAGNJ634H2UTKD2GZARFTFI25LUPM4DVYYE7ED2ANQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.284678955 (81h36m25.982247292s) [noise=true , address=185.167.204.205:4025 , piece_id=ZA2UB66ENOGTO76JP7WNSEIDGTOHBFY4EJIXIHSE55R6XKVZOLQA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.317320976 (81h36m26.014889282s) [noise=true , address=185.167.204.205:4030 , piece_id=TX4HHLN6OEQ3W3OHJXTXBH7RMOSVN5ALRRIMCYG6GXUHRCEOWRXQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.349729960 (81h36m26.047298267s) [noise=true , address=81.171.14.70:4031 , piece_id=QWI3ZIKVOZORRAXQIQ5MQOUOGOJZGTVQUAB4Z74BPUV6JGNVG7YA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.398340652 (81h36m26.095908968s) [noise=true , address=81.171.14.70:4035 , piece_id=ZYJD6KKUQHEVQM753L3BOUVSWTRTGJ3QL4MPSRW32BYS7WI3NXCA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.402029483 (81h36m26.099597799s) [noise=true , address=81.171.14.70:4039 , piece_id=UMVZ2DBSTQZI5RX2SBXZ54TOVJVGO7YL6LEKMQKCJ2PTFLXCWWAQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.430547239 (81h36m26.128115546s) [noise=true , address=89.149.201.147:4009 , piece_id=AL4GPBPIDDM2MB3G5YL4OYZKF7HHWO3MN77EL2MNJEFVC4RT5PLQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.455320098 (81h36m26.152888414s) [noise=true , address=81.171.14.70:4046 , piece_id=GVDXLZDYTELC4W2WSYL7HKTRC6CKYAD4A2HSBEA5JVVFFAFCG7OA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.519495047 (81h36m26.217063383s) [noise=true , address=185.167.204.206:4048 , piece_id=TA6RZFQAJUCUBP6FO4UMDZH6MOEE3OODX3UMYAD55HCUDX4RMSVQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.884448142 (81h36m26.582016447s) [noise=true , address=185.167.204.206:4046 , piece_id=CU257I2AOWKBX55AKS7FHWNY54IGZS7WS5UYMKASMXMTGCLWBGOQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.972874484 (81h36m26.6704428s) [noise=true , address=185.167.204.206:4072 , piece_id=LNCOWESTFDPRT5GMUZWCT6K2GTVRL7SY7EPMCL3BEU7D2ET36CMA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.105562604 (81h36m26.803130921s) [noise=true , address=185.167.204.205:4006 , piece_id=IUMA7QWVD7YY6YYOQ7TZAB5WNOKZSSEVG6S7RFYGOSAPGN3GAUXA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.110292050 (81h36m26.807860356s) [noise=true , address=89.149.201.147:4011 , piece_id=743ZJO643CQ6DWYJ7I5WWF3U35CEPHDISZVSK2NZKOUKJDOA27FA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.118428710 (81h36m26.815997016s) [noise=true , address=185.167.204.206:4063 , piece_id=6EQHKKLC2WL23KZ6C64343Q7UVYZVCSDGG3IW72VI77IHHECC3NA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.150488523 (81h36m26.848056839s) [noise=true , address=185.167.204.206:4043 , piece_id=2KGX72ULTVLFVN75NLCCPGJWFGJGJCBHOG6GACIO7KAIO57HH6KQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.410051915 (81h36m27.107620231s) [noise=true , address=89.149.201.147:4006 , piece_id=DIG4CLXPEKHJ7NTU5FSXIQ7UF2AZRP3IS5VBQZDTCLYVNU7AGOQA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.883572548 (81h36m27.581140854s) [noise=true , address=81.171.14.70:4045 , piece_id=O76W2ER3TCW7NV7QKUT5G34VVHVV77U7FSTLNCPLMNZWCOENMWPQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:41.043353306 (81h36m28.740921612s) [noise=true , address=81.171.14.70:4044 , piece_id=57QPDI374TVOD7IQAMV6O537BZTYKDJTGUMV6V4DT5HIO6TSXJZA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:41.043368334 (81h36m28.74093664s) [seg_pos={0 0}, upload=71621]: Segment reached optimal threshold of 26 pieces. Dec 9 01:05:41.043535717 (81h36m28.741104033s) [noise=true , address=81.171.14.70:4047 , piece_id=XJPFYSDOHMK3276AVOBQEWPB5GX4RRR7SDQDNIEKERHD63XICRKQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:41.043544413 (81h36m28.741112729s) [noise=true , address=81.171.14.70:4047 , piece_id=XJPFYSDOHMK3276AVOBQEWPB5GX4RRR7SDQDNIEKERHD63XICRKQ, seg_pos={0 0}, upload=71621]: Long tail cancellation due to optimal threshold Dec 9 01:05:41.043839945 (81h36m28.741408261s) [noise=true , address=81.171.14.70:4030 , piece_id=NEMDFBBUGZH2UJ6QRN3AEULAMUWK3FIMVOVXFA5UDI6IIEK66F6Q, seg_pos={0 0}, upload=71621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:41.043849824 (81h36m28.74141814s) [noise=true , address=81.171.14.70:4030 , piece_id=NEMDFBBUGZH2UJ6QRN3AEULAMUWK3FIMVOVXFA5UDI6IIEK66F6Q, seg_pos={0 0}, upload=71621]: Long tail cancellation due to optimal threshold Dec 9 01:05:41.043869772 (81h36m28.741438088s) [seg_pos={0 0}, upload=71621]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:05:41.044069243 (81h36m28.741637559s) [upload=71621]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:41.044074694 (81h36m28.74164301s) [upload=71621]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:05:41.304967689 (81h36m29.002536005s) [upload=71621]: Done uploading segments... Dec 9 01:05:41.304981745 (81h36m29.002550061s) [upload=71621]: Upload finished. err: <nil> Duration: 7.571682646s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 466.688215ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:09:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 466.7017ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 12 out of 60 is failed (threshold=3) [.......X........X...XX.....XX..X...X.....X.X.......X...X....] duration: 5.002695707s trace: https://jaeger.dev.storj.io/trace/3f20fc47e98690b9 FAIL (2025-12-09 01:08:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== .167.204.206:4072 , piece_id=LNCOWESTFDPRT5GMUZWCT6K2GTVRL7SY7EPMCL3BEU7D2ET36CMA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.800352280 (81h36m24.497920596s) [noise=true , address=89.149.201.147:4006 , piece_id=DIG4CLXPEKHJ7NTU5FSXIQ7UF2AZRP3IS5VBQZDTCLYVNU7AGOQA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.801378487 (81h36m24.498946803s) [noise=true , address=81.171.14.70:4044 , piece_id=57QPDI374TVOD7IQAMV6O537BZTYKDJTGUMV6V4DT5HIO6TSXJZA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.801993136 (81h36m24.499561441s) [noise=true , address=185.167.204.206:4063 , piece_id=6EQHKKLC2WL23KZ6C64343Q7UVYZVCSDGG3IW72VI77IHHECC3NA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.803054798 (81h36m24.500623114s) [noise=true , address=81.171.14.70:4035 , piece_id=ZYJD6KKUQHEVQM753L3BOUVSWTRTGJ3QL4MPSRW32BYS7WI3NXCA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.804454994 (81h36m24.502023311s) [noise=true , address=89.149.201.147:4004 , piece_id=OSLAKDSTRU2HKYKMBVBR5PEXHPDX2JZ4YKJWFLZQHTLSYKBY6FGA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.806500335 (81h36m24.504068651s) [noise=true , address=81.171.14.70:4032 , piece_id=4YQ37JCLYWHN7NDROVCYAK7LJYAKG6LEFHJ3SUHLJ4XZK65GN77A, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.808007841 (81h36m24.505576157s) [noise=true , address=81.171.14.70:4039 , piece_id=UMVZ2DBSTQZI5RX2SBXZ54TOVJVGO7YL6LEKMQKCJ2PTFLXCWWAQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.810776653 (81h36m24.508344959s) [noise=true , address=89.149.201.147:4011 , piece_id=743ZJO643CQ6DWYJ7I5WWF3U35CEPHDISZVSK2NZKOUKJDOA27FA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.822630489 (81h36m24.520198825s) [noise=true , address=185.167.204.206:4051 , piece_id=ERNWXDJUTTJCTOLBJXN7GBF6ZHFLPA6NDQREYLXMKZSHKVYKVNGQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.824087431 (81h36m24.521655737s) [noise=true , address=185.167.204.206:4043 , piece_id=2KGX72ULTVLFVN75NLCCPGJWFGJGJCBHOG6GACIO7KAIO57HH6KQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:37.301186810 (81h36m24.998755126s) [upload=71621]: Next returned nil segment Dec 9 01:05:37.301200385 (81h36m24.998768691s) [upload=71621]: Waiting for error group managing segments... Dec 9 01:05:37.502191335 (81h36m25.199759652s) [noise=true , address=81.171.14.70:4032 , piece_id=4YQ37JCLYWHN7NDROVCYAK7LJYAKG6LEFHJ3SUHLJ4XZK65GN77A, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.524058687 (81h36m25.221627003s) [noise=true , address=81.171.14.70:4041 , piece_id=XKVCPTKZKNQ3GBHLHVPRSH6DXE6R4QGOSW34MYAI6FT7LCELQ5YQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.540043689 (81h36m25.237612005s) [noise=true , address=185.167.204.205:4013 , piece_id=TGYBDY5F4MYWOGX5E3TT2SF3WVQZFSYDYWXY4I6YA3IKK4LM6YXQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.559903370 (81h36m25.257471686s) [noise=true , address=89.149.201.147:4020 , piece_id=3V6MIKGBDVFHUEL7TC6JUW7P6IXTOR2JSFOYO7A3S2Q27QDWC5YQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.581565809 (81h36m25.279134125s) [noise=true , address=89.149.201.147:4004 , piece_id=OSLAKDSTRU2HKYKMBVBR5PEXHPDX2JZ4YKJWFLZQHTLSYKBY6FGA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.913526920 (81h36m25.611095256s) [noise=true , address=185.167.204.206:4051 , piece_id=ERNWXDJUTTJCTOLBJXN7GBF6ZHFLPA6NDQREYLXMKZSHKVYKVNGQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.076632645 (81h36m25.774200991s) [noise=true , address=81.171.14.70:4033 , piece_id=H65LLAS4JHUZGGMBIA5WC3XH4HYULEOQDY7HS7JKBIYFQFGSOFOQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.115906113 (81h36m25.813474449s) [noise=true , address=89.149.201.147:4018 , piece_id=A3YGMQFONONZSFRQTLZNIBFBNLJPL7JUQOSBR2WULNEE5UYFXTUQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.199498056 (81h36m25.897066392s) [noise=true , address=81.171.14.70:4037 , piece_id=PDBORKWMIIEAGNJ634H2UTKD2GZARFTFI25LUPM4DVYYE7ED2ANQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.284678955 (81h36m25.982247292s) [noise=true , address=185.167.204.205:4025 , piece_id=ZA2UB66ENOGTO76JP7WNSEIDGTOHBFY4EJIXIHSE55R6XKVZOLQA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.317320976 (81h36m26.014889282s) [noise=true , address=185.167.204.205:4030 , piece_id=TX4HHLN6OEQ3W3OHJXTXBH7RMOSVN5ALRRIMCYG6GXUHRCEOWRXQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.349729960 (81h36m26.047298267s) [noise=true , address=81.171.14.70:4031 , piece_id=QWI3ZIKVOZORRAXQIQ5MQOUOGOJZGTVQUAB4Z74BPUV6JGNVG7YA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.398340652 (81h36m26.095908968s) [noise=true , address=81.171.14.70:4035 , piece_id=ZYJD6KKUQHEVQM753L3BOUVSWTRTGJ3QL4MPSRW32BYS7WI3NXCA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.402029483 (81h36m26.099597799s) [noise=true , address=81.171.14.70:4039 , piece_id=UMVZ2DBSTQZI5RX2SBXZ54TOVJVGO7YL6LEKMQKCJ2PTFLXCWWAQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.430547239 (81h36m26.128115546s) [noise=true , address=89.149.201.147:4009 , piece_id=AL4GPBPIDDM2MB3G5YL4OYZKF7HHWO3MN77EL2MNJEFVC4RT5PLQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.455320098 (81h36m26.152888414s) [noise=true , address=81.171.14.70:4046 , piece_id=GVDXLZDYTELC4W2WSYL7HKTRC6CKYAD4A2HSBEA5JVVFFAFCG7OA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.519495047 (81h36m26.217063383s) [noise=true , address=185.167.204.206:4048 , piece_id=TA6RZFQAJUCUBP6FO4UMDZH6MOEE3OODX3UMYAD55HCUDX4RMSVQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.884448142 (81h36m26.582016447s) [noise=true , address=185.167.204.206:4046 , piece_id=CU257I2AOWKBX55AKS7FHWNY54IGZS7WS5UYMKASMXMTGCLWBGOQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.972874484 (81h36m26.6704428s) [noise=true , address=185.167.204.206:4072 , piece_id=LNCOWESTFDPRT5GMUZWCT6K2GTVRL7SY7EPMCL3BEU7D2ET36CMA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.105562604 (81h36m26.803130921s) [noise=true , address=185.167.204.205:4006 , piece_id=IUMA7QWVD7YY6YYOQ7TZAB5WNOKZSSEVG6S7RFYGOSAPGN3GAUXA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.110292050 (81h36m26.807860356s) [noise=true , address=89.149.201.147:4011 , piece_id=743ZJO643CQ6DWYJ7I5WWF3U35CEPHDISZVSK2NZKOUKJDOA27FA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.118428710 (81h36m26.815997016s) [noise=true , address=185.167.204.206:4063 , piece_id=6EQHKKLC2WL23KZ6C64343Q7UVYZVCSDGG3IW72VI77IHHECC3NA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.150488523 (81h36m26.848056839s) [noise=true , address=185.167.204.206:4043 , piece_id=2KGX72ULTVLFVN75NLCCPGJWFGJGJCBHOG6GACIO7KAIO57HH6KQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.410051915 (81h36m27.107620231s) [noise=true , address=89.149.201.147:4006 , piece_id=DIG4CLXPEKHJ7NTU5FSXIQ7UF2AZRP3IS5VBQZDTCLYVNU7AGOQA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.883572548 (81h36m27.581140854s) [noise=true , address=81.171.14.70:4045 , piece_id=O76W2ER3TCW7NV7QKUT5G34VVHVV77U7FSTLNCPLMNZWCOENMWPQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:41.043353306 (81h36m28.740921612s) [noise=true , address=81.171.14.70:4044 , piece_id=57QPDI374TVOD7IQAMV6O537BZTYKDJTGUMV6V4DT5HIO6TSXJZA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:41.043368334 (81h36m28.74093664s) [seg_pos={0 0}, upload=71621]: Segment reached optimal threshold of 26 pieces. Dec 9 01:05:41.043535717 (81h36m28.741104033s) [noise=true , address=81.171.14.70:4047 , piece_id=XJPFYSDOHMK3276AVOBQEWPB5GX4RRR7SDQDNIEKERHD63XICRKQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:41.043544413 (81h36m28.741112729s) [noise=true , address=81.171.14.70:4047 , piece_id=XJPFYSDOHMK3276AVOBQEWPB5GX4RRR7SDQDNIEKERHD63XICRKQ, seg_pos={0 0}, upload=71621]: Long tail cancellation due to optimal threshold Dec 9 01:05:41.043839945 (81h36m28.741408261s) [noise=true , address=81.171.14.70:4030 , piece_id=NEMDFBBUGZH2UJ6QRN3AEULAMUWK3FIMVOVXFA5UDI6IIEK66F6Q, seg_pos={0 0}, upload=71621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:41.043849824 (81h36m28.74141814s) [noise=true , address=81.171.14.70:4030 , piece_id=NEMDFBBUGZH2UJ6QRN3AEULAMUWK3FIMVOVXFA5UDI6IIEK66F6Q, seg_pos={0 0}, upload=71621]: Long tail cancellation due to optimal threshold Dec 9 01:05:41.043869772 (81h36m28.741438088s) [seg_pos={0 0}, upload=71621]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:05:41.044069243 (81h36m28.741637559s) [upload=71621]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:41.044074694 (81h36m28.74164301s) [upload=71621]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:05:41.304967689 (81h36m29.002536005s) [upload=71621]: Done uploading segments... Dec 9 01:05:41.304981745 (81h36m29.002550061s) [upload=71621]: Upload finished. err: <nil> Duration: 5.002716126s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 12 out of 60 is failed (threshold=3) [........X........X...XX.....XX..X...X.....X.X.......X...X...] duration: 22.344474707s trace: https://jaeger.dev.storj.io/trace/3f20fc47e98690b9 FAIL (2025-12-09 01:08:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== .167.204.206:4072 , piece_id=LNCOWESTFDPRT5GMUZWCT6K2GTVRL7SY7EPMCL3BEU7D2ET36CMA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.800352280 (81h36m24.497920596s) [noise=true , address=89.149.201.147:4006 , piece_id=DIG4CLXPEKHJ7NTU5FSXIQ7UF2AZRP3IS5VBQZDTCLYVNU7AGOQA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.801378487 (81h36m24.498946803s) [noise=true , address=81.171.14.70:4044 , piece_id=57QPDI374TVOD7IQAMV6O537BZTYKDJTGUMV6V4DT5HIO6TSXJZA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.801993136 (81h36m24.499561441s) [noise=true , address=185.167.204.206:4063 , piece_id=6EQHKKLC2WL23KZ6C64343Q7UVYZVCSDGG3IW72VI77IHHECC3NA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.803054798 (81h36m24.500623114s) [noise=true , address=81.171.14.70:4035 , piece_id=ZYJD6KKUQHEVQM753L3BOUVSWTRTGJ3QL4MPSRW32BYS7WI3NXCA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.804454994 (81h36m24.502023311s) [noise=true , address=89.149.201.147:4004 , piece_id=OSLAKDSTRU2HKYKMBVBR5PEXHPDX2JZ4YKJWFLZQHTLSYKBY6FGA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.806500335 (81h36m24.504068651s) [noise=true , address=81.171.14.70:4032 , piece_id=4YQ37JCLYWHN7NDROVCYAK7LJYAKG6LEFHJ3SUHLJ4XZK65GN77A, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.808007841 (81h36m24.505576157s) [noise=true , address=81.171.14.70:4039 , piece_id=UMVZ2DBSTQZI5RX2SBXZ54TOVJVGO7YL6LEKMQKCJ2PTFLXCWWAQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.810776653 (81h36m24.508344959s) [noise=true , address=89.149.201.147:4011 , piece_id=743ZJO643CQ6DWYJ7I5WWF3U35CEPHDISZVSK2NZKOUKJDOA27FA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.822630489 (81h36m24.520198825s) [noise=true , address=185.167.204.206:4051 , piece_id=ERNWXDJUTTJCTOLBJXN7GBF6ZHFLPA6NDQREYLXMKZSHKVYKVNGQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.824087431 (81h36m24.521655737s) [noise=true , address=185.167.204.206:4043 , piece_id=2KGX72ULTVLFVN75NLCCPGJWFGJGJCBHOG6GACIO7KAIO57HH6KQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:37.301186810 (81h36m24.998755126s) [upload=71621]: Next returned nil segment Dec 9 01:05:37.301200385 (81h36m24.998768691s) [upload=71621]: Waiting for error group managing segments... Dec 9 01:05:37.502191335 (81h36m25.199759652s) [noise=true , address=81.171.14.70:4032 , piece_id=4YQ37JCLYWHN7NDROVCYAK7LJYAKG6LEFHJ3SUHLJ4XZK65GN77A, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.524058687 (81h36m25.221627003s) [noise=true , address=81.171.14.70:4041 , piece_id=XKVCPTKZKNQ3GBHLHVPRSH6DXE6R4QGOSW34MYAI6FT7LCELQ5YQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.540043689 (81h36m25.237612005s) [noise=true , address=185.167.204.205:4013 , piece_id=TGYBDY5F4MYWOGX5E3TT2SF3WVQZFSYDYWXY4I6YA3IKK4LM6YXQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.559903370 (81h36m25.257471686s) [noise=true , address=89.149.201.147:4020 , piece_id=3V6MIKGBDVFHUEL7TC6JUW7P6IXTOR2JSFOYO7A3S2Q27QDWC5YQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.581565809 (81h36m25.279134125s) [noise=true , address=89.149.201.147:4004 , piece_id=OSLAKDSTRU2HKYKMBVBR5PEXHPDX2JZ4YKJWFLZQHTLSYKBY6FGA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.913526920 (81h36m25.611095256s) [noise=true , address=185.167.204.206:4051 , piece_id=ERNWXDJUTTJCTOLBJXN7GBF6ZHFLPA6NDQREYLXMKZSHKVYKVNGQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.076632645 (81h36m25.774200991s) [noise=true , address=81.171.14.70:4033 , piece_id=H65LLAS4JHUZGGMBIA5WC3XH4HYULEOQDY7HS7JKBIYFQFGSOFOQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.115906113 (81h36m25.813474449s) [noise=true , address=89.149.201.147:4018 , piece_id=A3YGMQFONONZSFRQTLZNIBFBNLJPL7JUQOSBR2WULNEE5UYFXTUQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.199498056 (81h36m25.897066392s) [noise=true , address=81.171.14.70:4037 , piece_id=PDBORKWMIIEAGNJ634H2UTKD2GZARFTFI25LUPM4DVYYE7ED2ANQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.284678955 (81h36m25.982247292s) [noise=true , address=185.167.204.205:4025 , piece_id=ZA2UB66ENOGTO76JP7WNSEIDGTOHBFY4EJIXIHSE55R6XKVZOLQA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.317320976 (81h36m26.014889282s) [noise=true , address=185.167.204.205:4030 , piece_id=TX4HHLN6OEQ3W3OHJXTXBH7RMOSVN5ALRRIMCYG6GXUHRCEOWRXQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.349729960 (81h36m26.047298267s) [noise=true , address=81.171.14.70:4031 , piece_id=QWI3ZIKVOZORRAXQIQ5MQOUOGOJZGTVQUAB4Z74BPUV6JGNVG7YA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.398340652 (81h36m26.095908968s) [noise=true , address=81.171.14.70:4035 , piece_id=ZYJD6KKUQHEVQM753L3BOUVSWTRTGJ3QL4MPSRW32BYS7WI3NXCA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.402029483 (81h36m26.099597799s) [noise=true , address=81.171.14.70:4039 , piece_id=UMVZ2DBSTQZI5RX2SBXZ54TOVJVGO7YL6LEKMQKCJ2PTFLXCWWAQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.430547239 (81h36m26.128115546s) [noise=true , address=89.149.201.147:4009 , piece_id=AL4GPBPIDDM2MB3G5YL4OYZKF7HHWO3MN77EL2MNJEFVC4RT5PLQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.455320098 (81h36m26.152888414s) [noise=true , address=81.171.14.70:4046 , piece_id=GVDXLZDYTELC4W2WSYL7HKTRC6CKYAD4A2HSBEA5JVVFFAFCG7OA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.519495047 (81h36m26.217063383s) [noise=true , address=185.167.204.206:4048 , piece_id=TA6RZFQAJUCUBP6FO4UMDZH6MOEE3OODX3UMYAD55HCUDX4RMSVQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.884448142 (81h36m26.582016447s) [noise=true , address=185.167.204.206:4046 , piece_id=CU257I2AOWKBX55AKS7FHWNY54IGZS7WS5UYMKASMXMTGCLWBGOQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.972874484 (81h36m26.6704428s) [noise=true , address=185.167.204.206:4072 , piece_id=LNCOWESTFDPRT5GMUZWCT6K2GTVRL7SY7EPMCL3BEU7D2ET36CMA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.105562604 (81h36m26.803130921s) [noise=true , address=185.167.204.205:4006 , piece_id=IUMA7QWVD7YY6YYOQ7TZAB5WNOKZSSEVG6S7RFYGOSAPGN3GAUXA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.110292050 (81h36m26.807860356s) [noise=true , address=89.149.201.147:4011 , piece_id=743ZJO643CQ6DWYJ7I5WWF3U35CEPHDISZVSK2NZKOUKJDOA27FA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.118428710 (81h36m26.815997016s) [noise=true , address=185.167.204.206:4063 , piece_id=6EQHKKLC2WL23KZ6C64343Q7UVYZVCSDGG3IW72VI77IHHECC3NA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.150488523 (81h36m26.848056839s) [noise=true , address=185.167.204.206:4043 , piece_id=2KGX72ULTVLFVN75NLCCPGJWFGJGJCBHOG6GACIO7KAIO57HH6KQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.410051915 (81h36m27.107620231s) [noise=true , address=89.149.201.147:4006 , piece_id=DIG4CLXPEKHJ7NTU5FSXIQ7UF2AZRP3IS5VBQZDTCLYVNU7AGOQA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.883572548 (81h36m27.581140854s) [noise=true , address=81.171.14.70:4045 , piece_id=O76W2ER3TCW7NV7QKUT5G34VVHVV77U7FSTLNCPLMNZWCOENMWPQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:41.043353306 (81h36m28.740921612s) [noise=true , address=81.171.14.70:4044 , piece_id=57QPDI374TVOD7IQAMV6O537BZTYKDJTGUMV6V4DT5HIO6TSXJZA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:41.043368334 (81h36m28.74093664s) [seg_pos={0 0}, upload=71621]: Segment reached optimal threshold of 26 pieces. Dec 9 01:05:41.043535717 (81h36m28.741104033s) [noise=true , address=81.171.14.70:4047 , piece_id=XJPFYSDOHMK3276AVOBQEWPB5GX4RRR7SDQDNIEKERHD63XICRKQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:41.043544413 (81h36m28.741112729s) [noise=true , address=81.171.14.70:4047 , piece_id=XJPFYSDOHMK3276AVOBQEWPB5GX4RRR7SDQDNIEKERHD63XICRKQ, seg_pos={0 0}, upload=71621]: Long tail cancellation due to optimal threshold Dec 9 01:05:41.043839945 (81h36m28.741408261s) [noise=true , address=81.171.14.70:4030 , piece_id=NEMDFBBUGZH2UJ6QRN3AEULAMUWK3FIMVOVXFA5UDI6IIEK66F6Q, seg_pos={0 0}, upload=71621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:41.043849824 (81h36m28.74141814s) [noise=true , address=81.171.14.70:4030 , piece_id=NEMDFBBUGZH2UJ6QRN3AEULAMUWK3FIMVOVXFA5UDI6IIEK66F6Q, seg_pos={0 0}, upload=71621]: Long tail cancellation due to optimal threshold Dec 9 01:05:41.043869772 (81h36m28.741438088s) [seg_pos={0 0}, upload=71621]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:05:41.044069243 (81h36m28.741637559s) [upload=71621]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:41.044074694 (81h36m28.74164301s) [upload=71621]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:05:41.304967689 (81h36m29.002536005s) [upload=71621]: Done uploading segments... Dec 9 01:05:41.304981745 (81h36m29.002550061s) [upload=71621]: Upload finished. err: <nil> Duration: 22.344555589s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 450.56579ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:07:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.585847ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 12 out of 60 is failed (threshold=3) [.........X........X...XX.....XX..X...X.....X.X.......X...X..] duration: 4.487374573s trace: https://jaeger.dev.storj.io/trace/3f20fc47e98690b9 FAIL (2025-12-09 01:07:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== .167.204.206:4072 , piece_id=LNCOWESTFDPRT5GMUZWCT6K2GTVRL7SY7EPMCL3BEU7D2ET36CMA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.800352280 (81h36m24.497920596s) [noise=true , address=89.149.201.147:4006 , piece_id=DIG4CLXPEKHJ7NTU5FSXIQ7UF2AZRP3IS5VBQZDTCLYVNU7AGOQA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.801378487 (81h36m24.498946803s) [noise=true , address=81.171.14.70:4044 , piece_id=57QPDI374TVOD7IQAMV6O537BZTYKDJTGUMV6V4DT5HIO6TSXJZA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.801993136 (81h36m24.499561441s) [noise=true , address=185.167.204.206:4063 , piece_id=6EQHKKLC2WL23KZ6C64343Q7UVYZVCSDGG3IW72VI77IHHECC3NA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.803054798 (81h36m24.500623114s) [noise=true , address=81.171.14.70:4035 , piece_id=ZYJD6KKUQHEVQM753L3BOUVSWTRTGJ3QL4MPSRW32BYS7WI3NXCA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.804454994 (81h36m24.502023311s) [noise=true , address=89.149.201.147:4004 , piece_id=OSLAKDSTRU2HKYKMBVBR5PEXHPDX2JZ4YKJWFLZQHTLSYKBY6FGA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.806500335 (81h36m24.504068651s) [noise=true , address=81.171.14.70:4032 , piece_id=4YQ37JCLYWHN7NDROVCYAK7LJYAKG6LEFHJ3SUHLJ4XZK65GN77A, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.808007841 (81h36m24.505576157s) [noise=true , address=81.171.14.70:4039 , piece_id=UMVZ2DBSTQZI5RX2SBXZ54TOVJVGO7YL6LEKMQKCJ2PTFLXCWWAQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.810776653 (81h36m24.508344959s) [noise=true , address=89.149.201.147:4011 , piece_id=743ZJO643CQ6DWYJ7I5WWF3U35CEPHDISZVSK2NZKOUKJDOA27FA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.822630489 (81h36m24.520198825s) [noise=true , address=185.167.204.206:4051 , piece_id=ERNWXDJUTTJCTOLBJXN7GBF6ZHFLPA6NDQREYLXMKZSHKVYKVNGQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.824087431 (81h36m24.521655737s) [noise=true , address=185.167.204.206:4043 , piece_id=2KGX72ULTVLFVN75NLCCPGJWFGJGJCBHOG6GACIO7KAIO57HH6KQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:37.301186810 (81h36m24.998755126s) [upload=71621]: Next returned nil segment Dec 9 01:05:37.301200385 (81h36m24.998768691s) [upload=71621]: Waiting for error group managing segments... Dec 9 01:05:37.502191335 (81h36m25.199759652s) [noise=true , address=81.171.14.70:4032 , piece_id=4YQ37JCLYWHN7NDROVCYAK7LJYAKG6LEFHJ3SUHLJ4XZK65GN77A, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.524058687 (81h36m25.221627003s) [noise=true , address=81.171.14.70:4041 , piece_id=XKVCPTKZKNQ3GBHLHVPRSH6DXE6R4QGOSW34MYAI6FT7LCELQ5YQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.540043689 (81h36m25.237612005s) [noise=true , address=185.167.204.205:4013 , piece_id=TGYBDY5F4MYWOGX5E3TT2SF3WVQZFSYDYWXY4I6YA3IKK4LM6YXQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.559903370 (81h36m25.257471686s) [noise=true , address=89.149.201.147:4020 , piece_id=3V6MIKGBDVFHUEL7TC6JUW7P6IXTOR2JSFOYO7A3S2Q27QDWC5YQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.581565809 (81h36m25.279134125s) [noise=true , address=89.149.201.147:4004 , piece_id=OSLAKDSTRU2HKYKMBVBR5PEXHPDX2JZ4YKJWFLZQHTLSYKBY6FGA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.913526920 (81h36m25.611095256s) [noise=true , address=185.167.204.206:4051 , piece_id=ERNWXDJUTTJCTOLBJXN7GBF6ZHFLPA6NDQREYLXMKZSHKVYKVNGQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.076632645 (81h36m25.774200991s) [noise=true , address=81.171.14.70:4033 , piece_id=H65LLAS4JHUZGGMBIA5WC3XH4HYULEOQDY7HS7JKBIYFQFGSOFOQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.115906113 (81h36m25.813474449s) [noise=true , address=89.149.201.147:4018 , piece_id=A3YGMQFONONZSFRQTLZNIBFBNLJPL7JUQOSBR2WULNEE5UYFXTUQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.199498056 (81h36m25.897066392s) [noise=true , address=81.171.14.70:4037 , piece_id=PDBORKWMIIEAGNJ634H2UTKD2GZARFTFI25LUPM4DVYYE7ED2ANQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.284678955 (81h36m25.982247292s) [noise=true , address=185.167.204.205:4025 , piece_id=ZA2UB66ENOGTO76JP7WNSEIDGTOHBFY4EJIXIHSE55R6XKVZOLQA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.317320976 (81h36m26.014889282s) [noise=true , address=185.167.204.205:4030 , piece_id=TX4HHLN6OEQ3W3OHJXTXBH7RMOSVN5ALRRIMCYG6GXUHRCEOWRXQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.349729960 (81h36m26.047298267s) [noise=true , address=81.171.14.70:4031 , piece_id=QWI3ZIKVOZORRAXQIQ5MQOUOGOJZGTVQUAB4Z74BPUV6JGNVG7YA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.398340652 (81h36m26.095908968s) [noise=true , address=81.171.14.70:4035 , piece_id=ZYJD6KKUQHEVQM753L3BOUVSWTRTGJ3QL4MPSRW32BYS7WI3NXCA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.402029483 (81h36m26.099597799s) [noise=true , address=81.171.14.70:4039 , piece_id=UMVZ2DBSTQZI5RX2SBXZ54TOVJVGO7YL6LEKMQKCJ2PTFLXCWWAQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.430547239 (81h36m26.128115546s) [noise=true , address=89.149.201.147:4009 , piece_id=AL4GPBPIDDM2MB3G5YL4OYZKF7HHWO3MN77EL2MNJEFVC4RT5PLQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.455320098 (81h36m26.152888414s) [noise=true , address=81.171.14.70:4046 , piece_id=GVDXLZDYTELC4W2WSYL7HKTRC6CKYAD4A2HSBEA5JVVFFAFCG7OA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.519495047 (81h36m26.217063383s) [noise=true , address=185.167.204.206:4048 , piece_id=TA6RZFQAJUCUBP6FO4UMDZH6MOEE3OODX3UMYAD55HCUDX4RMSVQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.884448142 (81h36m26.582016447s) [noise=true , address=185.167.204.206:4046 , piece_id=CU257I2AOWKBX55AKS7FHWNY54IGZS7WS5UYMKASMXMTGCLWBGOQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.972874484 (81h36m26.6704428s) [noise=true , address=185.167.204.206:4072 , piece_id=LNCOWESTFDPRT5GMUZWCT6K2GTVRL7SY7EPMCL3BEU7D2ET36CMA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.105562604 (81h36m26.803130921s) [noise=true , address=185.167.204.205:4006 , piece_id=IUMA7QWVD7YY6YYOQ7TZAB5WNOKZSSEVG6S7RFYGOSAPGN3GAUXA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.110292050 (81h36m26.807860356s) [noise=true , address=89.149.201.147:4011 , piece_id=743ZJO643CQ6DWYJ7I5WWF3U35CEPHDISZVSK2NZKOUKJDOA27FA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.118428710 (81h36m26.815997016s) [noise=true , address=185.167.204.206:4063 , piece_id=6EQHKKLC2WL23KZ6C64343Q7UVYZVCSDGG3IW72VI77IHHECC3NA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.150488523 (81h36m26.848056839s) [noise=true , address=185.167.204.206:4043 , piece_id=2KGX72ULTVLFVN75NLCCPGJWFGJGJCBHOG6GACIO7KAIO57HH6KQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.410051915 (81h36m27.107620231s) [noise=true , address=89.149.201.147:4006 , piece_id=DIG4CLXPEKHJ7NTU5FSXIQ7UF2AZRP3IS5VBQZDTCLYVNU7AGOQA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.883572548 (81h36m27.581140854s) [noise=true , address=81.171.14.70:4045 , piece_id=O76W2ER3TCW7NV7QKUT5G34VVHVV77U7FSTLNCPLMNZWCOENMWPQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:41.043353306 (81h36m28.740921612s) [noise=true , address=81.171.14.70:4044 , piece_id=57QPDI374TVOD7IQAMV6O537BZTYKDJTGUMV6V4DT5HIO6TSXJZA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:41.043368334 (81h36m28.74093664s) [seg_pos={0 0}, upload=71621]: Segment reached optimal threshold of 26 pieces. Dec 9 01:05:41.043535717 (81h36m28.741104033s) [noise=true , address=81.171.14.70:4047 , piece_id=XJPFYSDOHMK3276AVOBQEWPB5GX4RRR7SDQDNIEKERHD63XICRKQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:41.043544413 (81h36m28.741112729s) [noise=true , address=81.171.14.70:4047 , piece_id=XJPFYSDOHMK3276AVOBQEWPB5GX4RRR7SDQDNIEKERHD63XICRKQ, seg_pos={0 0}, upload=71621]: Long tail cancellation due to optimal threshold Dec 9 01:05:41.043839945 (81h36m28.741408261s) [noise=true , address=81.171.14.70:4030 , piece_id=NEMDFBBUGZH2UJ6QRN3AEULAMUWK3FIMVOVXFA5UDI6IIEK66F6Q, seg_pos={0 0}, upload=71621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:41.043849824 (81h36m28.74141814s) [noise=true , address=81.171.14.70:4030 , piece_id=NEMDFBBUGZH2UJ6QRN3AEULAMUWK3FIMVOVXFA5UDI6IIEK66F6Q, seg_pos={0 0}, upload=71621]: Long tail cancellation due to optimal threshold Dec 9 01:05:41.043869772 (81h36m28.741438088s) [seg_pos={0 0}, upload=71621]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:05:41.044069243 (81h36m28.741637559s) [upload=71621]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:41.044074694 (81h36m28.74164301s) [upload=71621]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:05:41.304967689 (81h36m29.002536005s) [upload=71621]: Done uploading segments... Dec 9 01:05:41.304981745 (81h36m29.002550061s) [upload=71621]: Upload finished. err: <nil> Duration: 4.487396995s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 525.884177ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:06:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 525.89654ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 13 out of 60 is failed (threshold=3) [X.........X........X...XX.....XX..X...X.....X.X.......X...X.] duration: 6.402599708s trace: https://jaeger.dev.storj.io/trace/3f20fc47e98690b9 FAIL (2025-12-09 01:06:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== .167.204.206:4072 , piece_id=LNCOWESTFDPRT5GMUZWCT6K2GTVRL7SY7EPMCL3BEU7D2ET36CMA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.800352280 (81h36m24.497920596s) [noise=true , address=89.149.201.147:4006 , piece_id=DIG4CLXPEKHJ7NTU5FSXIQ7UF2AZRP3IS5VBQZDTCLYVNU7AGOQA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.801378487 (81h36m24.498946803s) [noise=true , address=81.171.14.70:4044 , piece_id=57QPDI374TVOD7IQAMV6O537BZTYKDJTGUMV6V4DT5HIO6TSXJZA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.801993136 (81h36m24.499561441s) [noise=true , address=185.167.204.206:4063 , piece_id=6EQHKKLC2WL23KZ6C64343Q7UVYZVCSDGG3IW72VI77IHHECC3NA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.803054798 (81h36m24.500623114s) [noise=true , address=81.171.14.70:4035 , piece_id=ZYJD6KKUQHEVQM753L3BOUVSWTRTGJ3QL4MPSRW32BYS7WI3NXCA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.804454994 (81h36m24.502023311s) [noise=true , address=89.149.201.147:4004 , piece_id=OSLAKDSTRU2HKYKMBVBR5PEXHPDX2JZ4YKJWFLZQHTLSYKBY6FGA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.806500335 (81h36m24.504068651s) [noise=true , address=81.171.14.70:4032 , piece_id=4YQ37JCLYWHN7NDROVCYAK7LJYAKG6LEFHJ3SUHLJ4XZK65GN77A, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.808007841 (81h36m24.505576157s) [noise=true , address=81.171.14.70:4039 , piece_id=UMVZ2DBSTQZI5RX2SBXZ54TOVJVGO7YL6LEKMQKCJ2PTFLXCWWAQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.810776653 (81h36m24.508344959s) [noise=true , address=89.149.201.147:4011 , piece_id=743ZJO643CQ6DWYJ7I5WWF3U35CEPHDISZVSK2NZKOUKJDOA27FA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.822630489 (81h36m24.520198825s) [noise=true , address=185.167.204.206:4051 , piece_id=ERNWXDJUTTJCTOLBJXN7GBF6ZHFLPA6NDQREYLXMKZSHKVYKVNGQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.824087431 (81h36m24.521655737s) [noise=true , address=185.167.204.206:4043 , piece_id=2KGX72ULTVLFVN75NLCCPGJWFGJGJCBHOG6GACIO7KAIO57HH6KQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:37.301186810 (81h36m24.998755126s) [upload=71621]: Next returned nil segment Dec 9 01:05:37.301200385 (81h36m24.998768691s) [upload=71621]: Waiting for error group managing segments... Dec 9 01:05:37.502191335 (81h36m25.199759652s) [noise=true , address=81.171.14.70:4032 , piece_id=4YQ37JCLYWHN7NDROVCYAK7LJYAKG6LEFHJ3SUHLJ4XZK65GN77A, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.524058687 (81h36m25.221627003s) [noise=true , address=81.171.14.70:4041 , piece_id=XKVCPTKZKNQ3GBHLHVPRSH6DXE6R4QGOSW34MYAI6FT7LCELQ5YQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.540043689 (81h36m25.237612005s) [noise=true , address=185.167.204.205:4013 , piece_id=TGYBDY5F4MYWOGX5E3TT2SF3WVQZFSYDYWXY4I6YA3IKK4LM6YXQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.559903370 (81h36m25.257471686s) [noise=true , address=89.149.201.147:4020 , piece_id=3V6MIKGBDVFHUEL7TC6JUW7P6IXTOR2JSFOYO7A3S2Q27QDWC5YQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.581565809 (81h36m25.279134125s) [noise=true , address=89.149.201.147:4004 , piece_id=OSLAKDSTRU2HKYKMBVBR5PEXHPDX2JZ4YKJWFLZQHTLSYKBY6FGA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.913526920 (81h36m25.611095256s) [noise=true , address=185.167.204.206:4051 , piece_id=ERNWXDJUTTJCTOLBJXN7GBF6ZHFLPA6NDQREYLXMKZSHKVYKVNGQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.076632645 (81h36m25.774200991s) [noise=true , address=81.171.14.70:4033 , piece_id=H65LLAS4JHUZGGMBIA5WC3XH4HYULEOQDY7HS7JKBIYFQFGSOFOQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.115906113 (81h36m25.813474449s) [noise=true , address=89.149.201.147:4018 , piece_id=A3YGMQFONONZSFRQTLZNIBFBNLJPL7JUQOSBR2WULNEE5UYFXTUQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.199498056 (81h36m25.897066392s) [noise=true , address=81.171.14.70:4037 , piece_id=PDBORKWMIIEAGNJ634H2UTKD2GZARFTFI25LUPM4DVYYE7ED2ANQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.284678955 (81h36m25.982247292s) [noise=true , address=185.167.204.205:4025 , piece_id=ZA2UB66ENOGTO76JP7WNSEIDGTOHBFY4EJIXIHSE55R6XKVZOLQA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.317320976 (81h36m26.014889282s) [noise=true , address=185.167.204.205:4030 , piece_id=TX4HHLN6OEQ3W3OHJXTXBH7RMOSVN5ALRRIMCYG6GXUHRCEOWRXQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.349729960 (81h36m26.047298267s) [noise=true , address=81.171.14.70:4031 , piece_id=QWI3ZIKVOZORRAXQIQ5MQOUOGOJZGTVQUAB4Z74BPUV6JGNVG7YA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.398340652 (81h36m26.095908968s) [noise=true , address=81.171.14.70:4035 , piece_id=ZYJD6KKUQHEVQM753L3BOUVSWTRTGJ3QL4MPSRW32BYS7WI3NXCA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.402029483 (81h36m26.099597799s) [noise=true , address=81.171.14.70:4039 , piece_id=UMVZ2DBSTQZI5RX2SBXZ54TOVJVGO7YL6LEKMQKCJ2PTFLXCWWAQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.430547239 (81h36m26.128115546s) [noise=true , address=89.149.201.147:4009 , piece_id=AL4GPBPIDDM2MB3G5YL4OYZKF7HHWO3MN77EL2MNJEFVC4RT5PLQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.455320098 (81h36m26.152888414s) [noise=true , address=81.171.14.70:4046 , piece_id=GVDXLZDYTELC4W2WSYL7HKTRC6CKYAD4A2HSBEA5JVVFFAFCG7OA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.519495047 (81h36m26.217063383s) [noise=true , address=185.167.204.206:4048 , piece_id=TA6RZFQAJUCUBP6FO4UMDZH6MOEE3OODX3UMYAD55HCUDX4RMSVQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.884448142 (81h36m26.582016447s) [noise=true , address=185.167.204.206:4046 , piece_id=CU257I2AOWKBX55AKS7FHWNY54IGZS7WS5UYMKASMXMTGCLWBGOQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.972874484 (81h36m26.6704428s) [noise=true , address=185.167.204.206:4072 , piece_id=LNCOWESTFDPRT5GMUZWCT6K2GTVRL7SY7EPMCL3BEU7D2ET36CMA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.105562604 (81h36m26.803130921s) [noise=true , address=185.167.204.205:4006 , piece_id=IUMA7QWVD7YY6YYOQ7TZAB5WNOKZSSEVG6S7RFYGOSAPGN3GAUXA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.110292050 (81h36m26.807860356s) [noise=true , address=89.149.201.147:4011 , piece_id=743ZJO643CQ6DWYJ7I5WWF3U35CEPHDISZVSK2NZKOUKJDOA27FA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.118428710 (81h36m26.815997016s) [noise=true , address=185.167.204.206:4063 , piece_id=6EQHKKLC2WL23KZ6C64343Q7UVYZVCSDGG3IW72VI77IHHECC3NA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.150488523 (81h36m26.848056839s) [noise=true , address=185.167.204.206:4043 , piece_id=2KGX72ULTVLFVN75NLCCPGJWFGJGJCBHOG6GACIO7KAIO57HH6KQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.410051915 (81h36m27.107620231s) [noise=true , address=89.149.201.147:4006 , piece_id=DIG4CLXPEKHJ7NTU5FSXIQ7UF2AZRP3IS5VBQZDTCLYVNU7AGOQA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.883572548 (81h36m27.581140854s) [noise=true , address=81.171.14.70:4045 , piece_id=O76W2ER3TCW7NV7QKUT5G34VVHVV77U7FSTLNCPLMNZWCOENMWPQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:41.043353306 (81h36m28.740921612s) [noise=true , address=81.171.14.70:4044 , piece_id=57QPDI374TVOD7IQAMV6O537BZTYKDJTGUMV6V4DT5HIO6TSXJZA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:41.043368334 (81h36m28.74093664s) [seg_pos={0 0}, upload=71621]: Segment reached optimal threshold of 26 pieces. Dec 9 01:05:41.043535717 (81h36m28.741104033s) [noise=true , address=81.171.14.70:4047 , piece_id=XJPFYSDOHMK3276AVOBQEWPB5GX4RRR7SDQDNIEKERHD63XICRKQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:41.043544413 (81h36m28.741112729s) [noise=true , address=81.171.14.70:4047 , piece_id=XJPFYSDOHMK3276AVOBQEWPB5GX4RRR7SDQDNIEKERHD63XICRKQ, seg_pos={0 0}, upload=71621]: Long tail cancellation due to optimal threshold Dec 9 01:05:41.043839945 (81h36m28.741408261s) [noise=true , address=81.171.14.70:4030 , piece_id=NEMDFBBUGZH2UJ6QRN3AEULAMUWK3FIMVOVXFA5UDI6IIEK66F6Q, seg_pos={0 0}, upload=71621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:41.043849824 (81h36m28.74141814s) [noise=true , address=81.171.14.70:4030 , piece_id=NEMDFBBUGZH2UJ6QRN3AEULAMUWK3FIMVOVXFA5UDI6IIEK66F6Q, seg_pos={0 0}, upload=71621]: Long tail cancellation due to optimal threshold Dec 9 01:05:41.043869772 (81h36m28.741438088s) [seg_pos={0 0}, upload=71621]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:05:41.044069243 (81h36m28.741637559s) [upload=71621]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:41.044074694 (81h36m28.74164301s) [upload=71621]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:05:41.304967689 (81h36m29.002536005s) [upload=71621]: Done uploading segments... Dec 9 01:05:41.304981745 (81h36m29.002550061s) [upload=71621]: Upload finished. err: <nil> Duration: 6.402611781s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 13 out of 60 is failed (threshold=3) [.X.........X........X...XX.....XX..X...X.....X.X.......X...X] duration: 1m0.00283385s trace: https://jaeger.dev.storj.io/trace/3f20fc47e98690b9 FAIL (2025-12-09 01:06:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== .167.204.206:4072 , piece_id=LNCOWESTFDPRT5GMUZWCT6K2GTVRL7SY7EPMCL3BEU7D2ET36CMA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.800352280 (81h36m24.497920596s) [noise=true , address=89.149.201.147:4006 , piece_id=DIG4CLXPEKHJ7NTU5FSXIQ7UF2AZRP3IS5VBQZDTCLYVNU7AGOQA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.801378487 (81h36m24.498946803s) [noise=true , address=81.171.14.70:4044 , piece_id=57QPDI374TVOD7IQAMV6O537BZTYKDJTGUMV6V4DT5HIO6TSXJZA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.801993136 (81h36m24.499561441s) [noise=true , address=185.167.204.206:4063 , piece_id=6EQHKKLC2WL23KZ6C64343Q7UVYZVCSDGG3IW72VI77IHHECC3NA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.803054798 (81h36m24.500623114s) [noise=true , address=81.171.14.70:4035 , piece_id=ZYJD6KKUQHEVQM753L3BOUVSWTRTGJ3QL4MPSRW32BYS7WI3NXCA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.804454994 (81h36m24.502023311s) [noise=true , address=89.149.201.147:4004 , piece_id=OSLAKDSTRU2HKYKMBVBR5PEXHPDX2JZ4YKJWFLZQHTLSYKBY6FGA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.806500335 (81h36m24.504068651s) [noise=true , address=81.171.14.70:4032 , piece_id=4YQ37JCLYWHN7NDROVCYAK7LJYAKG6LEFHJ3SUHLJ4XZK65GN77A, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.808007841 (81h36m24.505576157s) [noise=true , address=81.171.14.70:4039 , piece_id=UMVZ2DBSTQZI5RX2SBXZ54TOVJVGO7YL6LEKMQKCJ2PTFLXCWWAQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.810776653 (81h36m24.508344959s) [noise=true , address=89.149.201.147:4011 , piece_id=743ZJO643CQ6DWYJ7I5WWF3U35CEPHDISZVSK2NZKOUKJDOA27FA, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.822630489 (81h36m24.520198825s) [noise=true , address=185.167.204.206:4051 , piece_id=ERNWXDJUTTJCTOLBJXN7GBF6ZHFLPA6NDQREYLXMKZSHKVYKVNGQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:36.824087431 (81h36m24.521655737s) [noise=true , address=185.167.204.206:4043 , piece_id=2KGX72ULTVLFVN75NLCCPGJWFGJGJCBHOG6GACIO7KAIO57HH6KQ, seg_pos={0 0}, upload=71621]: Uploading piece... Dec 9 01:05:37.301186810 (81h36m24.998755126s) [upload=71621]: Next returned nil segment Dec 9 01:05:37.301200385 (81h36m24.998768691s) [upload=71621]: Waiting for error group managing segments... Dec 9 01:05:37.502191335 (81h36m25.199759652s) [noise=true , address=81.171.14.70:4032 , piece_id=4YQ37JCLYWHN7NDROVCYAK7LJYAKG6LEFHJ3SUHLJ4XZK65GN77A, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.524058687 (81h36m25.221627003s) [noise=true , address=81.171.14.70:4041 , piece_id=XKVCPTKZKNQ3GBHLHVPRSH6DXE6R4QGOSW34MYAI6FT7LCELQ5YQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.540043689 (81h36m25.237612005s) [noise=true , address=185.167.204.205:4013 , piece_id=TGYBDY5F4MYWOGX5E3TT2SF3WVQZFSYDYWXY4I6YA3IKK4LM6YXQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.559903370 (81h36m25.257471686s) [noise=true , address=89.149.201.147:4020 , piece_id=3V6MIKGBDVFHUEL7TC6JUW7P6IXTOR2JSFOYO7A3S2Q27QDWC5YQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.581565809 (81h36m25.279134125s) [noise=true , address=89.149.201.147:4004 , piece_id=OSLAKDSTRU2HKYKMBVBR5PEXHPDX2JZ4YKJWFLZQHTLSYKBY6FGA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:37.913526920 (81h36m25.611095256s) [noise=true , address=185.167.204.206:4051 , piece_id=ERNWXDJUTTJCTOLBJXN7GBF6ZHFLPA6NDQREYLXMKZSHKVYKVNGQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.076632645 (81h36m25.774200991s) [noise=true , address=81.171.14.70:4033 , piece_id=H65LLAS4JHUZGGMBIA5WC3XH4HYULEOQDY7HS7JKBIYFQFGSOFOQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.115906113 (81h36m25.813474449s) [noise=true , address=89.149.201.147:4018 , piece_id=A3YGMQFONONZSFRQTLZNIBFBNLJPL7JUQOSBR2WULNEE5UYFXTUQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.199498056 (81h36m25.897066392s) [noise=true , address=81.171.14.70:4037 , piece_id=PDBORKWMIIEAGNJ634H2UTKD2GZARFTFI25LUPM4DVYYE7ED2ANQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.284678955 (81h36m25.982247292s) [noise=true , address=185.167.204.205:4025 , piece_id=ZA2UB66ENOGTO76JP7WNSEIDGTOHBFY4EJIXIHSE55R6XKVZOLQA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.317320976 (81h36m26.014889282s) [noise=true , address=185.167.204.205:4030 , piece_id=TX4HHLN6OEQ3W3OHJXTXBH7RMOSVN5ALRRIMCYG6GXUHRCEOWRXQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.349729960 (81h36m26.047298267s) [noise=true , address=81.171.14.70:4031 , piece_id=QWI3ZIKVOZORRAXQIQ5MQOUOGOJZGTVQUAB4Z74BPUV6JGNVG7YA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.398340652 (81h36m26.095908968s) [noise=true , address=81.171.14.70:4035 , piece_id=ZYJD6KKUQHEVQM753L3BOUVSWTRTGJ3QL4MPSRW32BYS7WI3NXCA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.402029483 (81h36m26.099597799s) [noise=true , address=81.171.14.70:4039 , piece_id=UMVZ2DBSTQZI5RX2SBXZ54TOVJVGO7YL6LEKMQKCJ2PTFLXCWWAQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.430547239 (81h36m26.128115546s) [noise=true , address=89.149.201.147:4009 , piece_id=AL4GPBPIDDM2MB3G5YL4OYZKF7HHWO3MN77EL2MNJEFVC4RT5PLQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.455320098 (81h36m26.152888414s) [noise=true , address=81.171.14.70:4046 , piece_id=GVDXLZDYTELC4W2WSYL7HKTRC6CKYAD4A2HSBEA5JVVFFAFCG7OA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.519495047 (81h36m26.217063383s) [noise=true , address=185.167.204.206:4048 , piece_id=TA6RZFQAJUCUBP6FO4UMDZH6MOEE3OODX3UMYAD55HCUDX4RMSVQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.884448142 (81h36m26.582016447s) [noise=true , address=185.167.204.206:4046 , piece_id=CU257I2AOWKBX55AKS7FHWNY54IGZS7WS5UYMKASMXMTGCLWBGOQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:38.972874484 (81h36m26.6704428s) [noise=true , address=185.167.204.206:4072 , piece_id=LNCOWESTFDPRT5GMUZWCT6K2GTVRL7SY7EPMCL3BEU7D2ET36CMA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.105562604 (81h36m26.803130921s) [noise=true , address=185.167.204.205:4006 , piece_id=IUMA7QWVD7YY6YYOQ7TZAB5WNOKZSSEVG6S7RFYGOSAPGN3GAUXA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.110292050 (81h36m26.807860356s) [noise=true , address=89.149.201.147:4011 , piece_id=743ZJO643CQ6DWYJ7I5WWF3U35CEPHDISZVSK2NZKOUKJDOA27FA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.118428710 (81h36m26.815997016s) [noise=true , address=185.167.204.206:4063 , piece_id=6EQHKKLC2WL23KZ6C64343Q7UVYZVCSDGG3IW72VI77IHHECC3NA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.150488523 (81h36m26.848056839s) [noise=true , address=185.167.204.206:4043 , piece_id=2KGX72ULTVLFVN75NLCCPGJWFGJGJCBHOG6GACIO7KAIO57HH6KQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.410051915 (81h36m27.107620231s) [noise=true , address=89.149.201.147:4006 , piece_id=DIG4CLXPEKHJ7NTU5FSXIQ7UF2AZRP3IS5VBQZDTCLYVNU7AGOQA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:39.883572548 (81h36m27.581140854s) [noise=true , address=81.171.14.70:4045 , piece_id=O76W2ER3TCW7NV7QKUT5G34VVHVV77U7FSTLNCPLMNZWCOENMWPQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:41.043353306 (81h36m28.740921612s) [noise=true , address=81.171.14.70:4044 , piece_id=57QPDI374TVOD7IQAMV6O537BZTYKDJTGUMV6V4DT5HIO6TSXJZA, seg_pos={0 0}, upload=71621]: Done uploading piece. err: <nil> Dec 9 01:05:41.043368334 (81h36m28.74093664s) [seg_pos={0 0}, upload=71621]: Segment reached optimal threshold of 26 pieces. Dec 9 01:05:41.043535717 (81h36m28.741104033s) [noise=true , address=81.171.14.70:4047 , piece_id=XJPFYSDOHMK3276AVOBQEWPB5GX4RRR7SDQDNIEKERHD63XICRKQ, seg_pos={0 0}, upload=71621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:41.043544413 (81h36m28.741112729s) [noise=true , address=81.171.14.70:4047 , piece_id=XJPFYSDOHMK3276AVOBQEWPB5GX4RRR7SDQDNIEKERHD63XICRKQ, seg_pos={0 0}, upload=71621]: Long tail cancellation due to optimal threshold Dec 9 01:05:41.043839945 (81h36m28.741408261s) [noise=true , address=81.171.14.70:4030 , piece_id=NEMDFBBUGZH2UJ6QRN3AEULAMUWK3FIMVOVXFA5UDI6IIEK66F6Q, seg_pos={0 0}, upload=71621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:41.043849824 (81h36m28.74141814s) [noise=true , address=81.171.14.70:4030 , piece_id=NEMDFBBUGZH2UJ6QRN3AEULAMUWK3FIMVOVXFA5UDI6IIEK66F6Q, seg_pos={0 0}, upload=71621]: Long tail cancellation due to optimal threshold Dec 9 01:05:41.043869772 (81h36m28.741438088s) [seg_pos={0 0}, upload=71621]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:05:41.044069243 (81h36m28.741637559s) [upload=71621]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:41.044074694 (81h36m28.74164301s) [upload=71621]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:05:41.304967689 (81h36m29.002536005s) [upload=71621]: Done uploading segments... Dec 9 01:05:41.304981745 (81h36m29.002550061s) [upload=71621]: Upload finished. err: <nil> Duration: 1m0.002858076s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 398.834738ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:06:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 398.851019ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 405.897836ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:05:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 405.928183ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 421.572068ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:04:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 421.591184ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 12 out of 60 is failed (threshold=3) [..X.........X........X...XX.....XX..X...X.....X.X.......X...] duration: 6.693400219s trace: https://jaeger.dev.storj.io/trace/20a78b810804a847 FAIL (2025-12-09 01:04:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== address=89.149.201.147:4019 , piece_id=Y7W46MK5AWXBCYIIHHF7IVCGXTWFWONS2BE755EDIX5E6AK5SI6A, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622183609 (81h33m41.319751916s) [noise=true , address=185.167.204.206:4049 , piece_id=HPQZ233ZSTMWOO2LJ54YIOLMA3TZLERYCT2WBLD7YUX3DKXJNAKA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622386930 (81h33m41.319955246s) [noise=true , address=81.171.14.70:4047 , piece_id=TTCLMKTNXVVMR273VE5GHA3XISFNV7MC7P7JGARX2MVNYXXRUVJA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622622559 (81h33m41.320190875s) [noise=true , address=81.171.14.70:4030 , piece_id=AM6PNVJ7AFJJXMM3SLC3HSFUYPP3M2LTISTI3QDWZRBTPRJEIHMA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622643499 (81h33m41.320211805s) [noise=true , address=185.167.204.206:4063 , piece_id=BT4ZWNMVZRKCVUKOS6PXZSATMTE6HYLZ62WCFSP7VJRR2UF7F6BA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622913453 (81h33m41.320481769s) [noise=true , address=81.171.14.70:4034 , piece_id=4NJPYQJQPSILTOU65CT2HDWQQUXSADKXH5OG267SIGAWITHWTSNQ, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.623168048 (81h33m41.320736355s) [noise=true , address=81.171.14.70:4031 , piece_id=QIQTMVNWK3YRTS44LQEXAFLL53LH5FDWX5ZNEH62SRGWYOUTTCWQ, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.623595126 (81h33m41.321163442s) [noise=true , address=89.149.201.147:4007 , piece_id=OY2F63ZMOTBJWHUZ4QJT6U5NRJM6K7KJYTN2WOZO42KJGQJ2X3SQ, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.624047501 (81h33m41.321615808s) [noise=true , address=185.167.204.206:4059 , piece_id=XNVB6M5BXHKI5P7N5X7CNQJVDGKGLA2KXSUVJYE7UMHAYDEIDHLA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.624455554 (81h33m41.322023859s) [noise=true , address=185.167.204.206:4041 , piece_id=SU3535BNBST7OYCV5HHYSX2WTP7QSGG6GBZ5JTOHZAGA7PFIM4ZA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.624842728 (81h33m41.322411034s) [noise=true , address=81.171.14.70:4037 , piece_id=U2IXSBQ3ZIXDGENJ3LHIVSE7PXXW5CTFEPUPEQNCHVD4BEDT2EFA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:54.122478774 (81h33m41.82004709s) [upload=71581]: Next returned nil segment Dec 9 01:02:54.122488552 (81h33m41.820056858s) [upload=71581]: Waiting for error group managing segments... Dec 9 01:02:54.321329546 (81h33m42.018897862s) [noise=true , address=185.167.204.205:4025 , piece_id=CT4GN4WBLENNW37F7RNWHJ5VH2FBLRSI2HQLCZF74UDDCOKRUG4A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.343991201 (81h33m42.041559517s) [noise=true , address=81.171.14.70:4034 , piece_id=4NJPYQJQPSILTOU65CT2HDWQQUXSADKXH5OG267SIGAWITHWTSNQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.370043690 (81h33m42.067612006s) [noise=true , address=89.149.201.147:4017 , piece_id=UEATEOXYGUYX3HVH7EEKVC5HD66WHCKHYPIXHBZP6BGUTAX2HR6A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.376661643 (81h33m42.07422996s) [noise=true , address=89.149.201.147:4007 , piece_id=OY2F63ZMOTBJWHUZ4QJT6U5NRJM6K7KJYTN2WOZO42KJGQJ2X3SQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.401708565 (81h33m42.099276881s) [noise=true , address=185.167.204.205:4028 , piece_id=UMO7SDGBXYMYT5RS4DLK6ETXLMUIZLKXCHZLHG3MBKI32A2CWF3A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.633089208 (81h33m42.330657514s) [noise=true , address=89.149.201.147:4018 , piece_id=H73S4DDS4QAWGEA7EHS4E6OJW7D5XNTWZTJJ3ADRKCIKLVRWDCAQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.741338514 (81h33m42.43890683s) [noise=true , address=185.167.204.205:4012 , piece_id=E5QDFENON4SD7DZHOTMFRKLUFKZZU7JUUAZRX7IQ4VGK2PGT5ADA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.753404244 (81h33m42.45097256s) [noise=true , address=81.171.14.70:4027 , piece_id=4FOCQMSCZLGR33FGFQGNDAVLOOLDK3API3WCI5RPXEQPJUNEDDUQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.764954051 (81h33m42.462522356s) [noise=true , address=185.167.204.206:4041 , piece_id=SU3535BNBST7OYCV5HHYSX2WTP7QSGG6GBZ5JTOHZAGA7PFIM4ZA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.807974328 (81h33m42.505542634s) [noise=true , address=185.167.204.205:4030 , piece_id=ADVYGLHVRBODRAG463TKJHAIC423EQ6DXA5G34YW4OWN4N3JESKQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.835061370 (81h33m42.532629677s) [noise=true , address=89.149.201.147:4022 , piece_id=R4GG57UNN6KH3CEW33JNDP5IAL7I3VJADRAIQN2NIRRSGC6G42AQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.878969397 (81h33m42.576537713s) [noise=true , address=81.171.14.70:4047 , piece_id=TTCLMKTNXVVMR273VE5GHA3XISFNV7MC7P7JGARX2MVNYXXRUVJA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.962073739 (81h33m42.659642055s) [noise=true , address=81.171.14.70:4033 , piece_id=USDTUURFJQT5UEUCWKBTMSVX53ETMRWQ5WUPUDQJJQLWJF4P7R4A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.075478103 (81h33m42.77304641s) [noise=true , address=89.149.201.147:4019 , piece_id=Y7W46MK5AWXBCYIIHHF7IVCGXTWFWONS2BE755EDIX5E6AK5SI6A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.173729252 (81h33m42.871297578s) [noise=true , address=81.171.14.70:4044 , piece_id=2HDJXOP5LKDKXC77PPKBTVZEQYQJU2EMHDVOF65EIK7UBEUZKLDQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.173788082 (81h33m42.871356409s) [noise=true , address=185.167.204.205:4026 , piece_id=EFYHPATLNN7HPXULRSVYYEYUGEHGISW2M5UFFSXWZBD7TALATCPA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.174799451 (81h33m42.872367767s) [noise=true , address=81.171.14.70:4031 , piece_id=QIQTMVNWK3YRTS44LQEXAFLL53LH5FDWX5ZNEH62SRGWYOUTTCWQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.201626178 (81h33m42.899194494s) [noise=true , address=81.171.14.70:4028 , piece_id=57Y2OFIN2VMEVFTD2CSFLNQ3PR2AEHUOCPA7ZFSFBDQSTPYLMRFQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.238382964 (81h33m42.93595128s) [noise=true , address=185.167.204.206:4059 , piece_id=XNVB6M5BXHKI5P7N5X7CNQJVDGKGLA2KXSUVJYE7UMHAYDEIDHLA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.304031315 (81h33m43.001599661s) [noise=true , address=185.167.204.205:4019 , piece_id=3RXI74N64VXMTGXBXTYBR26OR36VTC7GO42FSLOYOTFME76JP3KA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.314661474 (81h33m43.01222981s) [noise=true , address=81.171.14.70:4037 , piece_id=U2IXSBQ3ZIXDGENJ3LHIVSE7PXXW5CTFEPUPEQNCHVD4BEDT2EFA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.343214445 (81h33m43.040782791s) [noise=true , address=81.171.14.70:4030 , piece_id=AM6PNVJ7AFJJXMM3SLC3HSFUYPP3M2LTISTI3QDWZRBTPRJEIHMA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.414496960 (81h33m43.112065306s) [noise=true , address=185.167.204.206:4049 , piece_id=HPQZ233ZSTMWOO2LJ54YIOLMA3TZLERYCT2WBLD7YUX3DKXJNAKA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.456785622 (81h33m43.154353958s) [noise=true , address=185.167.204.206:4050 , piece_id=ECT53MBVJD27OEB4WSIJZE7YPWR6RFTFIK4UNWF2RFGFV6YAADKQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.636209623 (81h33m43.333777949s) [noise=true , address=89.149.201.147:4011 , piece_id=YDOLUESQIUSIRI46JIXMIVAL6SAHTOKI2RTPUT3C5VOAMNC7WPBA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.669421126 (81h33m43.366989462s) [noise=true , address=185.167.204.206:4062 , piece_id=OFO5RM7TUYMVY6K7O5WGLSFLBK3U4M773PHXFWVYQH57TARKNLPA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.669438008 (81h33m43.367006344s) [seg_pos={0 0}, upload=71581]: Segment reached optimal threshold of 26 pieces. Dec 9 01:02:55.669673428 (81h33m43.367241764s) [noise=true , address=185.167.204.206:4063 , piece_id=BT4ZWNMVZRKCVUKOS6PXZSATMTE6HYLZ62WCFSP7VJRR2UF7F6BA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U): protocol: expected piece hash; context canceled; EOF Dec 9 01:02:55.669683017 (81h33m43.367251352s) [noise=true , address=185.167.204.206:4063 , piece_id=BT4ZWNMVZRKCVUKOS6PXZSATMTE6HYLZ62WCFSP7VJRR2UF7F6BA, seg_pos={0 0}, upload=71581]: Long tail cancellation due to optimal threshold Dec 9 01:02:55.669978538 (81h33m43.367546885s) [noise=true , address=81.171.14.70:4036 , piece_id=V2M27QLG7R2TA7MCRW6L77ALBJBCBOCKHNL5B6SW5UQWECKNI2TA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TcRxUCpNQVo3KLtKPK1zc6NXR4AdRCvqH4gtxticRH5MrKhVq): protocol: context canceled; context canceled Dec 9 01:02:55.669991432 (81h33m43.367559768s) [noise=true , address=81.171.14.70:4036 , piece_id=V2M27QLG7R2TA7MCRW6L77ALBJBCBOCKHNL5B6SW5UQWECKNI2TA, seg_pos={0 0}, upload=71581]: Long tail cancellation due to optimal threshold Dec 9 01:02:55.670014154 (81h33m43.36758249s) [seg_pos={0 0}, upload=71581]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:02:55.670094966 (81h33m43.367663302s) [upload=71581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:02:55.670100326 (81h33m43.367668662s) [upload=71581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:02:55.816215641 (81h33m43.513783957s) [upload=71581]: Done uploading segments... Dec 9 01:02:55.816250074 (81h33m43.513818381s) [upload=71581]: Upload finished. err: <nil> Duration: 6.69342231s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 12 out of 60 is failed (threshold=3) [...X.........X........X...XX.....XX..X...X.....X.X.......X..] duration: 4.942673266s trace: https://jaeger.dev.storj.io/trace/20a78b810804a847 FAIL (2025-12-09 01:04:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== address=89.149.201.147:4019 , piece_id=Y7W46MK5AWXBCYIIHHF7IVCGXTWFWONS2BE755EDIX5E6AK5SI6A, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622183609 (81h33m41.319751916s) [noise=true , address=185.167.204.206:4049 , piece_id=HPQZ233ZSTMWOO2LJ54YIOLMA3TZLERYCT2WBLD7YUX3DKXJNAKA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622386930 (81h33m41.319955246s) [noise=true , address=81.171.14.70:4047 , piece_id=TTCLMKTNXVVMR273VE5GHA3XISFNV7MC7P7JGARX2MVNYXXRUVJA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622622559 (81h33m41.320190875s) [noise=true , address=81.171.14.70:4030 , piece_id=AM6PNVJ7AFJJXMM3SLC3HSFUYPP3M2LTISTI3QDWZRBTPRJEIHMA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622643499 (81h33m41.320211805s) [noise=true , address=185.167.204.206:4063 , piece_id=BT4ZWNMVZRKCVUKOS6PXZSATMTE6HYLZ62WCFSP7VJRR2UF7F6BA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622913453 (81h33m41.320481769s) [noise=true , address=81.171.14.70:4034 , piece_id=4NJPYQJQPSILTOU65CT2HDWQQUXSADKXH5OG267SIGAWITHWTSNQ, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.623168048 (81h33m41.320736355s) [noise=true , address=81.171.14.70:4031 , piece_id=QIQTMVNWK3YRTS44LQEXAFLL53LH5FDWX5ZNEH62SRGWYOUTTCWQ, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.623595126 (81h33m41.321163442s) [noise=true , address=89.149.201.147:4007 , piece_id=OY2F63ZMOTBJWHUZ4QJT6U5NRJM6K7KJYTN2WOZO42KJGQJ2X3SQ, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.624047501 (81h33m41.321615808s) [noise=true , address=185.167.204.206:4059 , piece_id=XNVB6M5BXHKI5P7N5X7CNQJVDGKGLA2KXSUVJYE7UMHAYDEIDHLA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.624455554 (81h33m41.322023859s) [noise=true , address=185.167.204.206:4041 , piece_id=SU3535BNBST7OYCV5HHYSX2WTP7QSGG6GBZ5JTOHZAGA7PFIM4ZA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.624842728 (81h33m41.322411034s) [noise=true , address=81.171.14.70:4037 , piece_id=U2IXSBQ3ZIXDGENJ3LHIVSE7PXXW5CTFEPUPEQNCHVD4BEDT2EFA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:54.122478774 (81h33m41.82004709s) [upload=71581]: Next returned nil segment Dec 9 01:02:54.122488552 (81h33m41.820056858s) [upload=71581]: Waiting for error group managing segments... Dec 9 01:02:54.321329546 (81h33m42.018897862s) [noise=true , address=185.167.204.205:4025 , piece_id=CT4GN4WBLENNW37F7RNWHJ5VH2FBLRSI2HQLCZF74UDDCOKRUG4A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.343991201 (81h33m42.041559517s) [noise=true , address=81.171.14.70:4034 , piece_id=4NJPYQJQPSILTOU65CT2HDWQQUXSADKXH5OG267SIGAWITHWTSNQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.370043690 (81h33m42.067612006s) [noise=true , address=89.149.201.147:4017 , piece_id=UEATEOXYGUYX3HVH7EEKVC5HD66WHCKHYPIXHBZP6BGUTAX2HR6A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.376661643 (81h33m42.07422996s) [noise=true , address=89.149.201.147:4007 , piece_id=OY2F63ZMOTBJWHUZ4QJT6U5NRJM6K7KJYTN2WOZO42KJGQJ2X3SQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.401708565 (81h33m42.099276881s) [noise=true , address=185.167.204.205:4028 , piece_id=UMO7SDGBXYMYT5RS4DLK6ETXLMUIZLKXCHZLHG3MBKI32A2CWF3A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.633089208 (81h33m42.330657514s) [noise=true , address=89.149.201.147:4018 , piece_id=H73S4DDS4QAWGEA7EHS4E6OJW7D5XNTWZTJJ3ADRKCIKLVRWDCAQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.741338514 (81h33m42.43890683s) [noise=true , address=185.167.204.205:4012 , piece_id=E5QDFENON4SD7DZHOTMFRKLUFKZZU7JUUAZRX7IQ4VGK2PGT5ADA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.753404244 (81h33m42.45097256s) [noise=true , address=81.171.14.70:4027 , piece_id=4FOCQMSCZLGR33FGFQGNDAVLOOLDK3API3WCI5RPXEQPJUNEDDUQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.764954051 (81h33m42.462522356s) [noise=true , address=185.167.204.206:4041 , piece_id=SU3535BNBST7OYCV5HHYSX2WTP7QSGG6GBZ5JTOHZAGA7PFIM4ZA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.807974328 (81h33m42.505542634s) [noise=true , address=185.167.204.205:4030 , piece_id=ADVYGLHVRBODRAG463TKJHAIC423EQ6DXA5G34YW4OWN4N3JESKQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.835061370 (81h33m42.532629677s) [noise=true , address=89.149.201.147:4022 , piece_id=R4GG57UNN6KH3CEW33JNDP5IAL7I3VJADRAIQN2NIRRSGC6G42AQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.878969397 (81h33m42.576537713s) [noise=true , address=81.171.14.70:4047 , piece_id=TTCLMKTNXVVMR273VE5GHA3XISFNV7MC7P7JGARX2MVNYXXRUVJA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.962073739 (81h33m42.659642055s) [noise=true , address=81.171.14.70:4033 , piece_id=USDTUURFJQT5UEUCWKBTMSVX53ETMRWQ5WUPUDQJJQLWJF4P7R4A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.075478103 (81h33m42.77304641s) [noise=true , address=89.149.201.147:4019 , piece_id=Y7W46MK5AWXBCYIIHHF7IVCGXTWFWONS2BE755EDIX5E6AK5SI6A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.173729252 (81h33m42.871297578s) [noise=true , address=81.171.14.70:4044 , piece_id=2HDJXOP5LKDKXC77PPKBTVZEQYQJU2EMHDVOF65EIK7UBEUZKLDQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.173788082 (81h33m42.871356409s) [noise=true , address=185.167.204.205:4026 , piece_id=EFYHPATLNN7HPXULRSVYYEYUGEHGISW2M5UFFSXWZBD7TALATCPA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.174799451 (81h33m42.872367767s) [noise=true , address=81.171.14.70:4031 , piece_id=QIQTMVNWK3YRTS44LQEXAFLL53LH5FDWX5ZNEH62SRGWYOUTTCWQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.201626178 (81h33m42.899194494s) [noise=true , address=81.171.14.70:4028 , piece_id=57Y2OFIN2VMEVFTD2CSFLNQ3PR2AEHUOCPA7ZFSFBDQSTPYLMRFQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.238382964 (81h33m42.93595128s) [noise=true , address=185.167.204.206:4059 , piece_id=XNVB6M5BXHKI5P7N5X7CNQJVDGKGLA2KXSUVJYE7UMHAYDEIDHLA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.304031315 (81h33m43.001599661s) [noise=true , address=185.167.204.205:4019 , piece_id=3RXI74N64VXMTGXBXTYBR26OR36VTC7GO42FSLOYOTFME76JP3KA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.314661474 (81h33m43.01222981s) [noise=true , address=81.171.14.70:4037 , piece_id=U2IXSBQ3ZIXDGENJ3LHIVSE7PXXW5CTFEPUPEQNCHVD4BEDT2EFA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.343214445 (81h33m43.040782791s) [noise=true , address=81.171.14.70:4030 , piece_id=AM6PNVJ7AFJJXMM3SLC3HSFUYPP3M2LTISTI3QDWZRBTPRJEIHMA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.414496960 (81h33m43.112065306s) [noise=true , address=185.167.204.206:4049 , piece_id=HPQZ233ZSTMWOO2LJ54YIOLMA3TZLERYCT2WBLD7YUX3DKXJNAKA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.456785622 (81h33m43.154353958s) [noise=true , address=185.167.204.206:4050 , piece_id=ECT53MBVJD27OEB4WSIJZE7YPWR6RFTFIK4UNWF2RFGFV6YAADKQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.636209623 (81h33m43.333777949s) [noise=true , address=89.149.201.147:4011 , piece_id=YDOLUESQIUSIRI46JIXMIVAL6SAHTOKI2RTPUT3C5VOAMNC7WPBA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.669421126 (81h33m43.366989462s) [noise=true , address=185.167.204.206:4062 , piece_id=OFO5RM7TUYMVY6K7O5WGLSFLBK3U4M773PHXFWVYQH57TARKNLPA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.669438008 (81h33m43.367006344s) [seg_pos={0 0}, upload=71581]: Segment reached optimal threshold of 26 pieces. Dec 9 01:02:55.669673428 (81h33m43.367241764s) [noise=true , address=185.167.204.206:4063 , piece_id=BT4ZWNMVZRKCVUKOS6PXZSATMTE6HYLZ62WCFSP7VJRR2UF7F6BA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U): protocol: expected piece hash; context canceled; EOF Dec 9 01:02:55.669683017 (81h33m43.367251352s) [noise=true , address=185.167.204.206:4063 , piece_id=BT4ZWNMVZRKCVUKOS6PXZSATMTE6HYLZ62WCFSP7VJRR2UF7F6BA, seg_pos={0 0}, upload=71581]: Long tail cancellation due to optimal threshold Dec 9 01:02:55.669978538 (81h33m43.367546885s) [noise=true , address=81.171.14.70:4036 , piece_id=V2M27QLG7R2TA7MCRW6L77ALBJBCBOCKHNL5B6SW5UQWECKNI2TA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TcRxUCpNQVo3KLtKPK1zc6NXR4AdRCvqH4gtxticRH5MrKhVq): protocol: context canceled; context canceled Dec 9 01:02:55.669991432 (81h33m43.367559768s) [noise=true , address=81.171.14.70:4036 , piece_id=V2M27QLG7R2TA7MCRW6L77ALBJBCBOCKHNL5B6SW5UQWECKNI2TA, seg_pos={0 0}, upload=71581]: Long tail cancellation due to optimal threshold Dec 9 01:02:55.670014154 (81h33m43.36758249s) [seg_pos={0 0}, upload=71581]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:02:55.670094966 (81h33m43.367663302s) [upload=71581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:02:55.670100326 (81h33m43.367668662s) [upload=71581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:02:55.816215641 (81h33m43.513783957s) [upload=71581]: Done uploading segments... Dec 9 01:02:55.816250074 (81h33m43.513818381s) [upload=71581]: Upload finished. err: <nil> Duration: 4.942693063s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 13 out of 60 is failed (threshold=3) [X...X.........X........X...XX.....XX..X...X.....X.X.......X.] duration: 22.628078527s trace: https://jaeger.dev.storj.io/trace/20a78b810804a847 FAIL (2025-12-09 01:03:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== address=89.149.201.147:4019 , piece_id=Y7W46MK5AWXBCYIIHHF7IVCGXTWFWONS2BE755EDIX5E6AK5SI6A, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622183609 (81h33m41.319751916s) [noise=true , address=185.167.204.206:4049 , piece_id=HPQZ233ZSTMWOO2LJ54YIOLMA3TZLERYCT2WBLD7YUX3DKXJNAKA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622386930 (81h33m41.319955246s) [noise=true , address=81.171.14.70:4047 , piece_id=TTCLMKTNXVVMR273VE5GHA3XISFNV7MC7P7JGARX2MVNYXXRUVJA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622622559 (81h33m41.320190875s) [noise=true , address=81.171.14.70:4030 , piece_id=AM6PNVJ7AFJJXMM3SLC3HSFUYPP3M2LTISTI3QDWZRBTPRJEIHMA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622643499 (81h33m41.320211805s) [noise=true , address=185.167.204.206:4063 , piece_id=BT4ZWNMVZRKCVUKOS6PXZSATMTE6HYLZ62WCFSP7VJRR2UF7F6BA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622913453 (81h33m41.320481769s) [noise=true , address=81.171.14.70:4034 , piece_id=4NJPYQJQPSILTOU65CT2HDWQQUXSADKXH5OG267SIGAWITHWTSNQ, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.623168048 (81h33m41.320736355s) [noise=true , address=81.171.14.70:4031 , piece_id=QIQTMVNWK3YRTS44LQEXAFLL53LH5FDWX5ZNEH62SRGWYOUTTCWQ, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.623595126 (81h33m41.321163442s) [noise=true , address=89.149.201.147:4007 , piece_id=OY2F63ZMOTBJWHUZ4QJT6U5NRJM6K7KJYTN2WOZO42KJGQJ2X3SQ, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.624047501 (81h33m41.321615808s) [noise=true , address=185.167.204.206:4059 , piece_id=XNVB6M5BXHKI5P7N5X7CNQJVDGKGLA2KXSUVJYE7UMHAYDEIDHLA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.624455554 (81h33m41.322023859s) [noise=true , address=185.167.204.206:4041 , piece_id=SU3535BNBST7OYCV5HHYSX2WTP7QSGG6GBZ5JTOHZAGA7PFIM4ZA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.624842728 (81h33m41.322411034s) [noise=true , address=81.171.14.70:4037 , piece_id=U2IXSBQ3ZIXDGENJ3LHIVSE7PXXW5CTFEPUPEQNCHVD4BEDT2EFA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:54.122478774 (81h33m41.82004709s) [upload=71581]: Next returned nil segment Dec 9 01:02:54.122488552 (81h33m41.820056858s) [upload=71581]: Waiting for error group managing segments... Dec 9 01:02:54.321329546 (81h33m42.018897862s) [noise=true , address=185.167.204.205:4025 , piece_id=CT4GN4WBLENNW37F7RNWHJ5VH2FBLRSI2HQLCZF74UDDCOKRUG4A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.343991201 (81h33m42.041559517s) [noise=true , address=81.171.14.70:4034 , piece_id=4NJPYQJQPSILTOU65CT2HDWQQUXSADKXH5OG267SIGAWITHWTSNQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.370043690 (81h33m42.067612006s) [noise=true , address=89.149.201.147:4017 , piece_id=UEATEOXYGUYX3HVH7EEKVC5HD66WHCKHYPIXHBZP6BGUTAX2HR6A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.376661643 (81h33m42.07422996s) [noise=true , address=89.149.201.147:4007 , piece_id=OY2F63ZMOTBJWHUZ4QJT6U5NRJM6K7KJYTN2WOZO42KJGQJ2X3SQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.401708565 (81h33m42.099276881s) [noise=true , address=185.167.204.205:4028 , piece_id=UMO7SDGBXYMYT5RS4DLK6ETXLMUIZLKXCHZLHG3MBKI32A2CWF3A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.633089208 (81h33m42.330657514s) [noise=true , address=89.149.201.147:4018 , piece_id=H73S4DDS4QAWGEA7EHS4E6OJW7D5XNTWZTJJ3ADRKCIKLVRWDCAQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.741338514 (81h33m42.43890683s) [noise=true , address=185.167.204.205:4012 , piece_id=E5QDFENON4SD7DZHOTMFRKLUFKZZU7JUUAZRX7IQ4VGK2PGT5ADA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.753404244 (81h33m42.45097256s) [noise=true , address=81.171.14.70:4027 , piece_id=4FOCQMSCZLGR33FGFQGNDAVLOOLDK3API3WCI5RPXEQPJUNEDDUQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.764954051 (81h33m42.462522356s) [noise=true , address=185.167.204.206:4041 , piece_id=SU3535BNBST7OYCV5HHYSX2WTP7QSGG6GBZ5JTOHZAGA7PFIM4ZA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.807974328 (81h33m42.505542634s) [noise=true , address=185.167.204.205:4030 , piece_id=ADVYGLHVRBODRAG463TKJHAIC423EQ6DXA5G34YW4OWN4N3JESKQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.835061370 (81h33m42.532629677s) [noise=true , address=89.149.201.147:4022 , piece_id=R4GG57UNN6KH3CEW33JNDP5IAL7I3VJADRAIQN2NIRRSGC6G42AQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.878969397 (81h33m42.576537713s) [noise=true , address=81.171.14.70:4047 , piece_id=TTCLMKTNXVVMR273VE5GHA3XISFNV7MC7P7JGARX2MVNYXXRUVJA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.962073739 (81h33m42.659642055s) [noise=true , address=81.171.14.70:4033 , piece_id=USDTUURFJQT5UEUCWKBTMSVX53ETMRWQ5WUPUDQJJQLWJF4P7R4A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.075478103 (81h33m42.77304641s) [noise=true , address=89.149.201.147:4019 , piece_id=Y7W46MK5AWXBCYIIHHF7IVCGXTWFWONS2BE755EDIX5E6AK5SI6A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.173729252 (81h33m42.871297578s) [noise=true , address=81.171.14.70:4044 , piece_id=2HDJXOP5LKDKXC77PPKBTVZEQYQJU2EMHDVOF65EIK7UBEUZKLDQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.173788082 (81h33m42.871356409s) [noise=true , address=185.167.204.205:4026 , piece_id=EFYHPATLNN7HPXULRSVYYEYUGEHGISW2M5UFFSXWZBD7TALATCPA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.174799451 (81h33m42.872367767s) [noise=true , address=81.171.14.70:4031 , piece_id=QIQTMVNWK3YRTS44LQEXAFLL53LH5FDWX5ZNEH62SRGWYOUTTCWQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.201626178 (81h33m42.899194494s) [noise=true , address=81.171.14.70:4028 , piece_id=57Y2OFIN2VMEVFTD2CSFLNQ3PR2AEHUOCPA7ZFSFBDQSTPYLMRFQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.238382964 (81h33m42.93595128s) [noise=true , address=185.167.204.206:4059 , piece_id=XNVB6M5BXHKI5P7N5X7CNQJVDGKGLA2KXSUVJYE7UMHAYDEIDHLA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.304031315 (81h33m43.001599661s) [noise=true , address=185.167.204.205:4019 , piece_id=3RXI74N64VXMTGXBXTYBR26OR36VTC7GO42FSLOYOTFME76JP3KA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.314661474 (81h33m43.01222981s) [noise=true , address=81.171.14.70:4037 , piece_id=U2IXSBQ3ZIXDGENJ3LHIVSE7PXXW5CTFEPUPEQNCHVD4BEDT2EFA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.343214445 (81h33m43.040782791s) [noise=true , address=81.171.14.70:4030 , piece_id=AM6PNVJ7AFJJXMM3SLC3HSFUYPP3M2LTISTI3QDWZRBTPRJEIHMA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.414496960 (81h33m43.112065306s) [noise=true , address=185.167.204.206:4049 , piece_id=HPQZ233ZSTMWOO2LJ54YIOLMA3TZLERYCT2WBLD7YUX3DKXJNAKA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.456785622 (81h33m43.154353958s) [noise=true , address=185.167.204.206:4050 , piece_id=ECT53MBVJD27OEB4WSIJZE7YPWR6RFTFIK4UNWF2RFGFV6YAADKQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.636209623 (81h33m43.333777949s) [noise=true , address=89.149.201.147:4011 , piece_id=YDOLUESQIUSIRI46JIXMIVAL6SAHTOKI2RTPUT3C5VOAMNC7WPBA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.669421126 (81h33m43.366989462s) [noise=true , address=185.167.204.206:4062 , piece_id=OFO5RM7TUYMVY6K7O5WGLSFLBK3U4M773PHXFWVYQH57TARKNLPA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.669438008 (81h33m43.367006344s) [seg_pos={0 0}, upload=71581]: Segment reached optimal threshold of 26 pieces. Dec 9 01:02:55.669673428 (81h33m43.367241764s) [noise=true , address=185.167.204.206:4063 , piece_id=BT4ZWNMVZRKCVUKOS6PXZSATMTE6HYLZ62WCFSP7VJRR2UF7F6BA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U): protocol: expected piece hash; context canceled; EOF Dec 9 01:02:55.669683017 (81h33m43.367251352s) [noise=true , address=185.167.204.206:4063 , piece_id=BT4ZWNMVZRKCVUKOS6PXZSATMTE6HYLZ62WCFSP7VJRR2UF7F6BA, seg_pos={0 0}, upload=71581]: Long tail cancellation due to optimal threshold Dec 9 01:02:55.669978538 (81h33m43.367546885s) [noise=true , address=81.171.14.70:4036 , piece_id=V2M27QLG7R2TA7MCRW6L77ALBJBCBOCKHNL5B6SW5UQWECKNI2TA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TcRxUCpNQVo3KLtKPK1zc6NXR4AdRCvqH4gtxticRH5MrKhVq): protocol: context canceled; context canceled Dec 9 01:02:55.669991432 (81h33m43.367559768s) [noise=true , address=81.171.14.70:4036 , piece_id=V2M27QLG7R2TA7MCRW6L77ALBJBCBOCKHNL5B6SW5UQWECKNI2TA, seg_pos={0 0}, upload=71581]: Long tail cancellation due to optimal threshold Dec 9 01:02:55.670014154 (81h33m43.36758249s) [seg_pos={0 0}, upload=71581]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:02:55.670094966 (81h33m43.367663302s) [upload=71581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:02:55.670100326 (81h33m43.367668662s) [upload=71581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:02:55.816215641 (81h33m43.513783957s) [upload=71581]: Done uploading segments... Dec 9 01:02:55.816250074 (81h33m43.513818381s) [upload=71581]: Upload finished. err: <nil> Duration: 22.628091652s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 482.33442ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:03:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 482.348677ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 13 out of 60 is failed (threshold=3) [.X...X.........X........X...XX.....XX..X...X.....X.X.......X] duration: 33.006507172s trace: https://jaeger.dev.storj.io/trace/20a78b810804a847 FAIL (2025-12-09 01:03:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== address=89.149.201.147:4019 , piece_id=Y7W46MK5AWXBCYIIHHF7IVCGXTWFWONS2BE755EDIX5E6AK5SI6A, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622183609 (81h33m41.319751916s) [noise=true , address=185.167.204.206:4049 , piece_id=HPQZ233ZSTMWOO2LJ54YIOLMA3TZLERYCT2WBLD7YUX3DKXJNAKA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622386930 (81h33m41.319955246s) [noise=true , address=81.171.14.70:4047 , piece_id=TTCLMKTNXVVMR273VE5GHA3XISFNV7MC7P7JGARX2MVNYXXRUVJA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622622559 (81h33m41.320190875s) [noise=true , address=81.171.14.70:4030 , piece_id=AM6PNVJ7AFJJXMM3SLC3HSFUYPP3M2LTISTI3QDWZRBTPRJEIHMA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622643499 (81h33m41.320211805s) [noise=true , address=185.167.204.206:4063 , piece_id=BT4ZWNMVZRKCVUKOS6PXZSATMTE6HYLZ62WCFSP7VJRR2UF7F6BA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.622913453 (81h33m41.320481769s) [noise=true , address=81.171.14.70:4034 , piece_id=4NJPYQJQPSILTOU65CT2HDWQQUXSADKXH5OG267SIGAWITHWTSNQ, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.623168048 (81h33m41.320736355s) [noise=true , address=81.171.14.70:4031 , piece_id=QIQTMVNWK3YRTS44LQEXAFLL53LH5FDWX5ZNEH62SRGWYOUTTCWQ, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.623595126 (81h33m41.321163442s) [noise=true , address=89.149.201.147:4007 , piece_id=OY2F63ZMOTBJWHUZ4QJT6U5NRJM6K7KJYTN2WOZO42KJGQJ2X3SQ, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.624047501 (81h33m41.321615808s) [noise=true , address=185.167.204.206:4059 , piece_id=XNVB6M5BXHKI5P7N5X7CNQJVDGKGLA2KXSUVJYE7UMHAYDEIDHLA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.624455554 (81h33m41.322023859s) [noise=true , address=185.167.204.206:4041 , piece_id=SU3535BNBST7OYCV5HHYSX2WTP7QSGG6GBZ5JTOHZAGA7PFIM4ZA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:53.624842728 (81h33m41.322411034s) [noise=true , address=81.171.14.70:4037 , piece_id=U2IXSBQ3ZIXDGENJ3LHIVSE7PXXW5CTFEPUPEQNCHVD4BEDT2EFA, seg_pos={0 0}, upload=71581]: Uploading piece... Dec 9 01:02:54.122478774 (81h33m41.82004709s) [upload=71581]: Next returned nil segment Dec 9 01:02:54.122488552 (81h33m41.820056858s) [upload=71581]: Waiting for error group managing segments... Dec 9 01:02:54.321329546 (81h33m42.018897862s) [noise=true , address=185.167.204.205:4025 , piece_id=CT4GN4WBLENNW37F7RNWHJ5VH2FBLRSI2HQLCZF74UDDCOKRUG4A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.343991201 (81h33m42.041559517s) [noise=true , address=81.171.14.70:4034 , piece_id=4NJPYQJQPSILTOU65CT2HDWQQUXSADKXH5OG267SIGAWITHWTSNQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.370043690 (81h33m42.067612006s) [noise=true , address=89.149.201.147:4017 , piece_id=UEATEOXYGUYX3HVH7EEKVC5HD66WHCKHYPIXHBZP6BGUTAX2HR6A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.376661643 (81h33m42.07422996s) [noise=true , address=89.149.201.147:4007 , piece_id=OY2F63ZMOTBJWHUZ4QJT6U5NRJM6K7KJYTN2WOZO42KJGQJ2X3SQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.401708565 (81h33m42.099276881s) [noise=true , address=185.167.204.205:4028 , piece_id=UMO7SDGBXYMYT5RS4DLK6ETXLMUIZLKXCHZLHG3MBKI32A2CWF3A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.633089208 (81h33m42.330657514s) [noise=true , address=89.149.201.147:4018 , piece_id=H73S4DDS4QAWGEA7EHS4E6OJW7D5XNTWZTJJ3ADRKCIKLVRWDCAQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.741338514 (81h33m42.43890683s) [noise=true , address=185.167.204.205:4012 , piece_id=E5QDFENON4SD7DZHOTMFRKLUFKZZU7JUUAZRX7IQ4VGK2PGT5ADA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.753404244 (81h33m42.45097256s) [noise=true , address=81.171.14.70:4027 , piece_id=4FOCQMSCZLGR33FGFQGNDAVLOOLDK3API3WCI5RPXEQPJUNEDDUQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.764954051 (81h33m42.462522356s) [noise=true , address=185.167.204.206:4041 , piece_id=SU3535BNBST7OYCV5HHYSX2WTP7QSGG6GBZ5JTOHZAGA7PFIM4ZA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.807974328 (81h33m42.505542634s) [noise=true , address=185.167.204.205:4030 , piece_id=ADVYGLHVRBODRAG463TKJHAIC423EQ6DXA5G34YW4OWN4N3JESKQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.835061370 (81h33m42.532629677s) [noise=true , address=89.149.201.147:4022 , piece_id=R4GG57UNN6KH3CEW33JNDP5IAL7I3VJADRAIQN2NIRRSGC6G42AQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.878969397 (81h33m42.576537713s) [noise=true , address=81.171.14.70:4047 , piece_id=TTCLMKTNXVVMR273VE5GHA3XISFNV7MC7P7JGARX2MVNYXXRUVJA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:54.962073739 (81h33m42.659642055s) [noise=true , address=81.171.14.70:4033 , piece_id=USDTUURFJQT5UEUCWKBTMSVX53ETMRWQ5WUPUDQJJQLWJF4P7R4A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.075478103 (81h33m42.77304641s) [noise=true , address=89.149.201.147:4019 , piece_id=Y7W46MK5AWXBCYIIHHF7IVCGXTWFWONS2BE755EDIX5E6AK5SI6A, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.173729252 (81h33m42.871297578s) [noise=true , address=81.171.14.70:4044 , piece_id=2HDJXOP5LKDKXC77PPKBTVZEQYQJU2EMHDVOF65EIK7UBEUZKLDQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.173788082 (81h33m42.871356409s) [noise=true , address=185.167.204.205:4026 , piece_id=EFYHPATLNN7HPXULRSVYYEYUGEHGISW2M5UFFSXWZBD7TALATCPA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.174799451 (81h33m42.872367767s) [noise=true , address=81.171.14.70:4031 , piece_id=QIQTMVNWK3YRTS44LQEXAFLL53LH5FDWX5ZNEH62SRGWYOUTTCWQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.201626178 (81h33m42.899194494s) [noise=true , address=81.171.14.70:4028 , piece_id=57Y2OFIN2VMEVFTD2CSFLNQ3PR2AEHUOCPA7ZFSFBDQSTPYLMRFQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.238382964 (81h33m42.93595128s) [noise=true , address=185.167.204.206:4059 , piece_id=XNVB6M5BXHKI5P7N5X7CNQJVDGKGLA2KXSUVJYE7UMHAYDEIDHLA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.304031315 (81h33m43.001599661s) [noise=true , address=185.167.204.205:4019 , piece_id=3RXI74N64VXMTGXBXTYBR26OR36VTC7GO42FSLOYOTFME76JP3KA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.314661474 (81h33m43.01222981s) [noise=true , address=81.171.14.70:4037 , piece_id=U2IXSBQ3ZIXDGENJ3LHIVSE7PXXW5CTFEPUPEQNCHVD4BEDT2EFA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.343214445 (81h33m43.040782791s) [noise=true , address=81.171.14.70:4030 , piece_id=AM6PNVJ7AFJJXMM3SLC3HSFUYPP3M2LTISTI3QDWZRBTPRJEIHMA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.414496960 (81h33m43.112065306s) [noise=true , address=185.167.204.206:4049 , piece_id=HPQZ233ZSTMWOO2LJ54YIOLMA3TZLERYCT2WBLD7YUX3DKXJNAKA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.456785622 (81h33m43.154353958s) [noise=true , address=185.167.204.206:4050 , piece_id=ECT53MBVJD27OEB4WSIJZE7YPWR6RFTFIK4UNWF2RFGFV6YAADKQ, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.636209623 (81h33m43.333777949s) [noise=true , address=89.149.201.147:4011 , piece_id=YDOLUESQIUSIRI46JIXMIVAL6SAHTOKI2RTPUT3C5VOAMNC7WPBA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.669421126 (81h33m43.366989462s) [noise=true , address=185.167.204.206:4062 , piece_id=OFO5RM7TUYMVY6K7O5WGLSFLBK3U4M773PHXFWVYQH57TARKNLPA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: <nil> Dec 9 01:02:55.669438008 (81h33m43.367006344s) [seg_pos={0 0}, upload=71581]: Segment reached optimal threshold of 26 pieces. Dec 9 01:02:55.669673428 (81h33m43.367241764s) [noise=true , address=185.167.204.206:4063 , piece_id=BT4ZWNMVZRKCVUKOS6PXZSATMTE6HYLZ62WCFSP7VJRR2UF7F6BA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U): protocol: expected piece hash; context canceled; EOF Dec 9 01:02:55.669683017 (81h33m43.367251352s) [noise=true , address=185.167.204.206:4063 , piece_id=BT4ZWNMVZRKCVUKOS6PXZSATMTE6HYLZ62WCFSP7VJRR2UF7F6BA, seg_pos={0 0}, upload=71581]: Long tail cancellation due to optimal threshold Dec 9 01:02:55.669978538 (81h33m43.367546885s) [noise=true , address=81.171.14.70:4036 , piece_id=V2M27QLG7R2TA7MCRW6L77ALBJBCBOCKHNL5B6SW5UQWECKNI2TA, seg_pos={0 0}, upload=71581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TcRxUCpNQVo3KLtKPK1zc6NXR4AdRCvqH4gtxticRH5MrKhVq): protocol: context canceled; context canceled Dec 9 01:02:55.669991432 (81h33m43.367559768s) [noise=true , address=81.171.14.70:4036 , piece_id=V2M27QLG7R2TA7MCRW6L77ALBJBCBOCKHNL5B6SW5UQWECKNI2TA, seg_pos={0 0}, upload=71581]: Long tail cancellation due to optimal threshold Dec 9 01:02:55.670014154 (81h33m43.36758249s) [seg_pos={0 0}, upload=71581]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 01:02:55.670094966 (81h33m43.367663302s) [upload=71581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:02:55.670100326 (81h33m43.367668662s) [upload=71581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:02:55.816215641 (81h33m43.513783957s) [upload=71581]: Done uploading segments... Dec 9 01:02:55.816250074 (81h33m43.513818381s) [upload=71581]: Upload finished. err: <nil> Duration: 33.006545735s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 399.505363ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:03:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 399.526784ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 402.198827ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:02:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 402.216489ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 12 out of 60 is failed (threshold=3) [..X...X.........X........X...XX.....XX..X...X.....X.X.......] duration: 4.853258742s trace: https://jaeger.dev.storj.io/trace/73cbc41c8163da51 FAIL (2025-12-09 01:02:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =89.149.201.147:4019 , piece_id=44XAWC4PX6JDYDNQES7RFJOL44BKSMZL6JXD4VSNSJ63EFCU6IGA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.653731184 (81h27m19.35129949s) [noise=true , address=89.149.201.147:4020 , piece_id=6UT7ISSBNWIIRYDAJ7AFVXYNPNB2AJD36BWEX5UBM5ZUXYNZZBOA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.654018120 (81h27m19.351586436s) [noise=true , address=81.171.14.70:4047 , piece_id=Y3IFJKBSGE47TODR4FTRYSCQFPXYDAQK6PIPO4ARTIC7CLR56S4Q, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.654327128 (81h27m19.351895434s) [noise=true , address=185.167.204.205:4032 , piece_id=ZRKSY5TYZ47BN5NJD734DAXN544U6TDJJNSYDVU7KPHEF2BTJAMA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.654782739 (81h27m19.352351055s) [noise=true , address=185.167.204.205:4015 , piece_id=D5IU7DJK5QUUXDZSJST7LMPOTQ43LMPPRQ3P46VKORYA7QWHRB2Q, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.655140286 (81h27m19.352708602s) [noise=true , address=185.167.204.205:4035 , piece_id=EKHXDUJ6JUKWPGLRZI5QJGUBF4QQKJDBSKF64H7RNVS6Q3UXOSHA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.655534964 (81h27m19.35310328s) [noise=true , address=185.167.204.206:4066 , piece_id=KW6JXLGMA66TJ2X3BSQ53MYERCPKIBT76PPBVE3PFIUMUKIKPBIA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.655963444 (81h27m19.35353175s) [noise=true , address=89.149.201.147:4010 , piece_id=RKCPZSXPW6PL3CJZMOWMWWC6NRU3MRLZMUFX5ZPFHJI55U7ZIP7Q, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.656381366 (81h27m19.353949672s) [noise=true , address=185.167.204.205:4030 , piece_id=2WEVN7VFRXBRV277D5W5B42L3WDGYT2V4WSJTKSTMD3WBKFUUVNQ, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.656743802 (81h27m19.354312108s) [noise=true , address=89.149.201.147:4021 , piece_id=QXEVXQODAXU2YMITYDM7VYD2ZKM5CRLEOKN5VRBTOLRJ6ZIVEYXQ, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.657206457 (81h27m19.354774763s) [noise=true , address=89.149.201.147:4006 , piece_id=KGZQZGUW3HO7HT5R3NX4NSKZTSB7ND537INBKOGAKFQGRBNHJ3TA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:32.160763915 (81h27m19.858332231s) [upload=71490]: Next returned nil segment Dec 9 00:56:32.160772261 (81h27m19.858340566s) [upload=71490]: Waiting for error group managing segments... Dec 9 00:56:32.367091113 (81h27m20.06465945s) [noise=true , address=185.167.204.205:4013 , piece_id=NNBGYPP6DGVX3FC4DZMC43ZAMHFPSZMY7KGRRIIGLFKCM3SQHHPA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.382359607 (81h27m20.079927943s) [noise=true , address=185.167.204.205:4009 , piece_id=QAL7FK7ZORITTNCJT4MNNFNPKNBQ54DXRCH2J6X6E4OEYOMAG5AQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.401869895 (81h27m20.099438232s) [noise=true , address=89.149.201.147:4004 , piece_id=63FVBOL7WODTHOLQFC75CWPSUQCJWS7LODV5FBRH2SICL3K3XWCQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.406066094 (81h27m20.103634431s) [noise=true , address=89.149.201.147:4021 , piece_id=QXEVXQODAXU2YMITYDM7VYD2ZKM5CRLEOKN5VRBTOLRJ6ZIVEYXQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.411331421 (81h27m20.108899797s) [noise=true , address=185.167.204.205:4015 , piece_id=D5IU7DJK5QUUXDZSJST7LMPOTQ43LMPPRQ3P46VKORYA7QWHRB2Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.418812968 (81h27m20.116381303s) [noise=true , address=89.149.201.147:4005 , piece_id=BYT4SLGEUH2AKLMHKBZ6QSQVJKOY2RVWQJC56BRV62QEKTBWKFRA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.420572304 (81h27m20.118140651s) [noise=true , address=81.171.14.70:4047 , piece_id=Y3IFJKBSGE47TODR4FTRYSCQFPXYDAQK6PIPO4ARTIC7CLR56S4Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.527718158 (81h27m20.225286494s) [noise=true , address=81.171.14.70:4044 , piece_id=PKZTY4JWVBCHCUHLL5MXXVWQ5MWWMUWR7I6SVQNI7BTR27IDPRRQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.574245548 (81h27m20.271813885s) [noise=true , address=185.167.204.206:4066 , piece_id=KW6JXLGMA66TJ2X3BSQ53MYERCPKIBT76PPBVE3PFIUMUKIKPBIA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.811440733 (81h27m20.509009049s) [noise=true , address=89.149.201.147:4015 , piece_id=EYC7EULWHAVARKFIMAQEDCMWHPTUSJBDFCYJT4RVSLSLYA4LV2CA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.814580629 (81h27m20.512148934s) [noise=true , address=89.149.201.147:4006 , piece_id=KGZQZGUW3HO7HT5R3NX4NSKZTSB7ND537INBKOGAKFQGRBNHJ3TA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.842364613 (81h27m20.539932918s) [noise=true , address=89.149.201.147:4017 , piece_id=H4BUDJVTVHDT7KYBJCBYXVHSHW6XVUICOHPAWFDWUCAFDRSLGX3Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.873751909 (81h27m20.571320235s) [noise=true , address=89.149.201.147:4019 , piece_id=44XAWC4PX6JDYDNQES7RFJOL44BKSMZL6JXD4VSNSJ63EFCU6IGA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.887103492 (81h27m20.584671828s) [noise=true , address=81.171.14.70:4025 , piece_id=AHLEQXU6UBZR2AANMXY72UGDKKCNW5RTDGD7MMQO4W2MMHUPQLUA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.908520581 (81h27m20.606088898s) [noise=true , address=185.167.204.205:4026 , piece_id=2W4DN6TY7AJJWQXYWSS6YZIGN563IXIQ2RMYWG3XHEL47PC7CJYQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.952690147 (81h27m20.650258454s) [noise=true , address=185.167.204.206:4049 , piece_id=WJOGRXIB3TD7JRCE7EZA6Q2JYL3SVHWKCIGBQ7JPBXSUFEOK4I3A, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.008828632 (81h27m20.706396977s) [noise=true , address=81.171.14.70:4038 , piece_id=BACFOGLKUZOOCDGQJIYHQVCDTUOGFGNAEEDZK246ULZSNFIYEOMA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.103114172 (81h27m20.800682508s) [noise=true , address=185.167.204.206:4070 , piece_id=ELEURNR5CXP7624QVG2M2ZK7HHZM2GHQXL5CPYSA2ODQAOXE3NKA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.134718753 (81h27m20.832287079s) [noise=true , address=81.171.14.70:4045 , piece_id=JYLMRVFZJQQMYNJVJ2GFM3QABGZZIPZCGNO4JVKOUISUSF2ZGGZQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.208666869 (81h27m20.906235185s) [noise=true , address=89.149.201.147:4020 , piece_id=6UT7ISSBNWIIRYDAJ7AFVXYNPNB2AJD36BWEX5UBM5ZUXYNZZBOA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.502522334 (81h27m21.20009065s) [noise=true , address=81.171.14.70:4040 , piece_id=LS33AXEKVAVNEVBJZ4XVVNIVW54Y3GCPI3RY4QPA5JNLJ32PBS5Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.520716964 (81h27m21.2182853s) [noise=true , address=89.149.201.147:4010 , piece_id=RKCPZSXPW6PL3CJZMOWMWWC6NRU3MRLZMUFX5ZPFHJI55U7ZIP7Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.541974847 (81h27m21.239543173s) [noise=true , address=185.167.204.205:4035 , piece_id=EKHXDUJ6JUKWPGLRZI5QJGUBF4QQKJDBSKF64H7RNVS6Q3UXOSHA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.901468573 (81h27m21.599036879s) [noise=true , address=185.167.204.205:4030 , piece_id=2WEVN7VFRXBRV277D5W5B42L3WDGYT2V4WSJTKSTMD3WBKFUUVNQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.908515979 (81h27m21.606084295s) [noise=true , address=185.167.204.205:4032 , piece_id=ZRKSY5TYZ47BN5NJD734DAXN544U6TDJJNSYDVU7KPHEF2BTJAMA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:34.121817866 (81h27m21.819386183s) [noise=true , address=81.171.14.70:4048 , piece_id=7W4TKSAVVTAZTOWVSI4PMDF57WOARYODCQOOYDBMTEUBDK26FM5A, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:34.121835138 (81h27m21.819403454s) [seg_pos={0 0}, upload=71490]: Segment reached optimal threshold of 26 pieces. Dec 9 00:56:34.122000397 (81h27m21.819568713s) [noise=true , address=89.149.201.147:4007 , piece_id=O52IHBKR3GBNALB7E2OCMVZJ6A4CXVNEPUDUQANAF2VATGZ5U46Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 00:56:34.122006168 (81h27m21.819574484s) [noise=true , address=89.149.201.147:4007 , piece_id=O52IHBKR3GBNALB7E2OCMVZJ6A4CXVNEPUDUQANAF2VATGZ5U46Q, seg_pos={0 0}, upload=71490]: Long tail cancellation due to optimal threshold Dec 9 00:56:34.122138234 (81h27m21.81970655s) [noise=true , address=81.171.14.70:4046 , piece_id=Q27COUPBROIQC4IVWVO55AJLV7BKYZACQI57NIDFOYNWLV5NITJQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Dec 9 00:56:34.122166547 (81h27m21.819734864s) [noise=true , address=81.171.14.70:4046 , piece_id=Q27COUPBROIQC4IVWVO55AJLV7BKYZACQI57NIDFOYNWLV5NITJQ, seg_pos={0 0}, upload=71490]: Long tail cancellation due to optimal threshold Dec 9 00:56:34.122190121 (81h27m21.819758438s) [seg_pos={0 0}, upload=71490]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 00:56:34.122256786 (81h27m21.819825102s) [upload=71490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 00:56:34.122261705 (81h27m21.819830011s) [upload=71490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 00:56:34.272548935 (81h27m21.970117241s) [upload=71490]: Done uploading segments... Dec 9 00:56:34.272567188 (81h27m21.970135495s) [upload=71490]: Upload finished. err: <nil> Duration: 4.853276334s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 12 out of 60 is failed (threshold=3) [...X...X.........X........X...XX.....XX..X...X.....X.X......] duration: 41.177831026s trace: https://jaeger.dev.storj.io/trace/73cbc41c8163da51 FAIL (2025-12-09 01:02:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =89.149.201.147:4019 , piece_id=44XAWC4PX6JDYDNQES7RFJOL44BKSMZL6JXD4VSNSJ63EFCU6IGA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.653731184 (81h27m19.35129949s) [noise=true , address=89.149.201.147:4020 , piece_id=6UT7ISSBNWIIRYDAJ7AFVXYNPNB2AJD36BWEX5UBM5ZUXYNZZBOA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.654018120 (81h27m19.351586436s) [noise=true , address=81.171.14.70:4047 , piece_id=Y3IFJKBSGE47TODR4FTRYSCQFPXYDAQK6PIPO4ARTIC7CLR56S4Q, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.654327128 (81h27m19.351895434s) [noise=true , address=185.167.204.205:4032 , piece_id=ZRKSY5TYZ47BN5NJD734DAXN544U6TDJJNSYDVU7KPHEF2BTJAMA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.654782739 (81h27m19.352351055s) [noise=true , address=185.167.204.205:4015 , piece_id=D5IU7DJK5QUUXDZSJST7LMPOTQ43LMPPRQ3P46VKORYA7QWHRB2Q, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.655140286 (81h27m19.352708602s) [noise=true , address=185.167.204.205:4035 , piece_id=EKHXDUJ6JUKWPGLRZI5QJGUBF4QQKJDBSKF64H7RNVS6Q3UXOSHA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.655534964 (81h27m19.35310328s) [noise=true , address=185.167.204.206:4066 , piece_id=KW6JXLGMA66TJ2X3BSQ53MYERCPKIBT76PPBVE3PFIUMUKIKPBIA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.655963444 (81h27m19.35353175s) [noise=true , address=89.149.201.147:4010 , piece_id=RKCPZSXPW6PL3CJZMOWMWWC6NRU3MRLZMUFX5ZPFHJI55U7ZIP7Q, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.656381366 (81h27m19.353949672s) [noise=true , address=185.167.204.205:4030 , piece_id=2WEVN7VFRXBRV277D5W5B42L3WDGYT2V4WSJTKSTMD3WBKFUUVNQ, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.656743802 (81h27m19.354312108s) [noise=true , address=89.149.201.147:4021 , piece_id=QXEVXQODAXU2YMITYDM7VYD2ZKM5CRLEOKN5VRBTOLRJ6ZIVEYXQ, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.657206457 (81h27m19.354774763s) [noise=true , address=89.149.201.147:4006 , piece_id=KGZQZGUW3HO7HT5R3NX4NSKZTSB7ND537INBKOGAKFQGRBNHJ3TA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:32.160763915 (81h27m19.858332231s) [upload=71490]: Next returned nil segment Dec 9 00:56:32.160772261 (81h27m19.858340566s) [upload=71490]: Waiting for error group managing segments... Dec 9 00:56:32.367091113 (81h27m20.06465945s) [noise=true , address=185.167.204.205:4013 , piece_id=NNBGYPP6DGVX3FC4DZMC43ZAMHFPSZMY7KGRRIIGLFKCM3SQHHPA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.382359607 (81h27m20.079927943s) [noise=true , address=185.167.204.205:4009 , piece_id=QAL7FK7ZORITTNCJT4MNNFNPKNBQ54DXRCH2J6X6E4OEYOMAG5AQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.401869895 (81h27m20.099438232s) [noise=true , address=89.149.201.147:4004 , piece_id=63FVBOL7WODTHOLQFC75CWPSUQCJWS7LODV5FBRH2SICL3K3XWCQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.406066094 (81h27m20.103634431s) [noise=true , address=89.149.201.147:4021 , piece_id=QXEVXQODAXU2YMITYDM7VYD2ZKM5CRLEOKN5VRBTOLRJ6ZIVEYXQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.411331421 (81h27m20.108899797s) [noise=true , address=185.167.204.205:4015 , piece_id=D5IU7DJK5QUUXDZSJST7LMPOTQ43LMPPRQ3P46VKORYA7QWHRB2Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.418812968 (81h27m20.116381303s) [noise=true , address=89.149.201.147:4005 , piece_id=BYT4SLGEUH2AKLMHKBZ6QSQVJKOY2RVWQJC56BRV62QEKTBWKFRA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.420572304 (81h27m20.118140651s) [noise=true , address=81.171.14.70:4047 , piece_id=Y3IFJKBSGE47TODR4FTRYSCQFPXYDAQK6PIPO4ARTIC7CLR56S4Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.527718158 (81h27m20.225286494s) [noise=true , address=81.171.14.70:4044 , piece_id=PKZTY4JWVBCHCUHLL5MXXVWQ5MWWMUWR7I6SVQNI7BTR27IDPRRQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.574245548 (81h27m20.271813885s) [noise=true , address=185.167.204.206:4066 , piece_id=KW6JXLGMA66TJ2X3BSQ53MYERCPKIBT76PPBVE3PFIUMUKIKPBIA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.811440733 (81h27m20.509009049s) [noise=true , address=89.149.201.147:4015 , piece_id=EYC7EULWHAVARKFIMAQEDCMWHPTUSJBDFCYJT4RVSLSLYA4LV2CA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.814580629 (81h27m20.512148934s) [noise=true , address=89.149.201.147:4006 , piece_id=KGZQZGUW3HO7HT5R3NX4NSKZTSB7ND537INBKOGAKFQGRBNHJ3TA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.842364613 (81h27m20.539932918s) [noise=true , address=89.149.201.147:4017 , piece_id=H4BUDJVTVHDT7KYBJCBYXVHSHW6XVUICOHPAWFDWUCAFDRSLGX3Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.873751909 (81h27m20.571320235s) [noise=true , address=89.149.201.147:4019 , piece_id=44XAWC4PX6JDYDNQES7RFJOL44BKSMZL6JXD4VSNSJ63EFCU6IGA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.887103492 (81h27m20.584671828s) [noise=true , address=81.171.14.70:4025 , piece_id=AHLEQXU6UBZR2AANMXY72UGDKKCNW5RTDGD7MMQO4W2MMHUPQLUA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.908520581 (81h27m20.606088898s) [noise=true , address=185.167.204.205:4026 , piece_id=2W4DN6TY7AJJWQXYWSS6YZIGN563IXIQ2RMYWG3XHEL47PC7CJYQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.952690147 (81h27m20.650258454s) [noise=true , address=185.167.204.206:4049 , piece_id=WJOGRXIB3TD7JRCE7EZA6Q2JYL3SVHWKCIGBQ7JPBXSUFEOK4I3A, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.008828632 (81h27m20.706396977s) [noise=true , address=81.171.14.70:4038 , piece_id=BACFOGLKUZOOCDGQJIYHQVCDTUOGFGNAEEDZK246ULZSNFIYEOMA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.103114172 (81h27m20.800682508s) [noise=true , address=185.167.204.206:4070 , piece_id=ELEURNR5CXP7624QVG2M2ZK7HHZM2GHQXL5CPYSA2ODQAOXE3NKA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.134718753 (81h27m20.832287079s) [noise=true , address=81.171.14.70:4045 , piece_id=JYLMRVFZJQQMYNJVJ2GFM3QABGZZIPZCGNO4JVKOUISUSF2ZGGZQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.208666869 (81h27m20.906235185s) [noise=true , address=89.149.201.147:4020 , piece_id=6UT7ISSBNWIIRYDAJ7AFVXYNPNB2AJD36BWEX5UBM5ZUXYNZZBOA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.502522334 (81h27m21.20009065s) [noise=true , address=81.171.14.70:4040 , piece_id=LS33AXEKVAVNEVBJZ4XVVNIVW54Y3GCPI3RY4QPA5JNLJ32PBS5Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.520716964 (81h27m21.2182853s) [noise=true , address=89.149.201.147:4010 , piece_id=RKCPZSXPW6PL3CJZMOWMWWC6NRU3MRLZMUFX5ZPFHJI55U7ZIP7Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.541974847 (81h27m21.239543173s) [noise=true , address=185.167.204.205:4035 , piece_id=EKHXDUJ6JUKWPGLRZI5QJGUBF4QQKJDBSKF64H7RNVS6Q3UXOSHA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.901468573 (81h27m21.599036879s) [noise=true , address=185.167.204.205:4030 , piece_id=2WEVN7VFRXBRV277D5W5B42L3WDGYT2V4WSJTKSTMD3WBKFUUVNQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.908515979 (81h27m21.606084295s) [noise=true , address=185.167.204.205:4032 , piece_id=ZRKSY5TYZ47BN5NJD734DAXN544U6TDJJNSYDVU7KPHEF2BTJAMA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:34.121817866 (81h27m21.819386183s) [noise=true , address=81.171.14.70:4048 , piece_id=7W4TKSAVVTAZTOWVSI4PMDF57WOARYODCQOOYDBMTEUBDK26FM5A, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:34.121835138 (81h27m21.819403454s) [seg_pos={0 0}, upload=71490]: Segment reached optimal threshold of 26 pieces. Dec 9 00:56:34.122000397 (81h27m21.819568713s) [noise=true , address=89.149.201.147:4007 , piece_id=O52IHBKR3GBNALB7E2OCMVZJ6A4CXVNEPUDUQANAF2VATGZ5U46Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 00:56:34.122006168 (81h27m21.819574484s) [noise=true , address=89.149.201.147:4007 , piece_id=O52IHBKR3GBNALB7E2OCMVZJ6A4CXVNEPUDUQANAF2VATGZ5U46Q, seg_pos={0 0}, upload=71490]: Long tail cancellation due to optimal threshold Dec 9 00:56:34.122138234 (81h27m21.81970655s) [noise=true , address=81.171.14.70:4046 , piece_id=Q27COUPBROIQC4IVWVO55AJLV7BKYZACQI57NIDFOYNWLV5NITJQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Dec 9 00:56:34.122166547 (81h27m21.819734864s) [noise=true , address=81.171.14.70:4046 , piece_id=Q27COUPBROIQC4IVWVO55AJLV7BKYZACQI57NIDFOYNWLV5NITJQ, seg_pos={0 0}, upload=71490]: Long tail cancellation due to optimal threshold Dec 9 00:56:34.122190121 (81h27m21.819758438s) [seg_pos={0 0}, upload=71490]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 00:56:34.122256786 (81h27m21.819825102s) [upload=71490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 00:56:34.122261705 (81h27m21.819830011s) [upload=71490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 00:56:34.272548935 (81h27m21.970117241s) [upload=71490]: Done uploading segments... Dec 9 00:56:34.272567188 (81h27m21.970135495s) [upload=71490]: Upload finished. err: <nil> Duration: 41.177854579s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 416.973125ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:01:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 416.990138ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 382.682617ms trace: https://jaeger.dev.storj.io/trace/4dbaa1759c71b1f4 FAIL (2025-12-09 01:01:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 00:41:54.622468710 (81h12m42.320037046s) [upload=71278]: Starting upload Dec 9 00:41:54.622543659 (81h12m42.320111995s) [upload=71278]: Done starting upload Dec 9 00:41:54.622571812 (81h12m42.320140128s) [upload=71278]: Uploading segments... Dec 9 00:41:54.622994281 (81h12m42.320562588s) [upload=71278]: Got next segment. Inline: false Dec 9 00:41:54.623028135 (81h12m42.320596451s) [upload=71278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 00:41:54.623032322 (81h12m42.320600629s) [upload=71278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 00:41:54.645317651 (81h12m42.342885968s) [seg_pos={0 0}, upload=71278]: Begin upload segment... Dec 9 00:41:54.645344462 (81h12m42.342912778s) [seg_pos={0 0}, upload=71278]: Done begin upload segment. Dec 9 00:41:54.645381130 (81h12m42.342949447s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645391058 (81h12m42.342959375s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.645865044 (81h12m42.34343336s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646379516 (81h12m42.343947832s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.646694834 (81h12m42.34426315s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647101132 (81h12m42.344669438s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647561132 (81h12m42.345129438s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647976397 (81h12m42.345544714s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.647983160 (81h12m42.345551476s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648389529 (81h12m42.345957845s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.648769520 (81h12m42.346337826s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650448076 (81h12m42.348016392s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.650815922 (81h12m42.348384228s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.651280570 (81h12m42.348848876s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652276761 (81h12m42.349845077s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.652581671 (81h12m42.350149977s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Uploading piece... Dec 9 00:41:54.690634333 (81h12m42.388202649s) [noise=true , address=77.90.31.19:4841 , piece_id=3DSCPC5E7M4NDL5RNHCROVN5IV3DFP7K35SE6QEC6FCP6STWUL6A, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.691392579 (81h12m42.388960895s) [noise=true , address=77.90.31.13:4626 , piece_id=B35MAB25LESGWCMN2VPVF5G745IV6GG7IUV5VYUXFM6PCRDMBMDQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 00:41:54.758795599 (81h12m42.456363906s) [upload=71278]: Next returned nil segment Dec 9 00:41:54.758816898 (81h12m42.456385204s) [upload=71278]: Waiting for error group managing segments... Dec 9 00:41:54.770879913 (81h12m42.468448219s) [noise=true , address=145.79.227.38:4979 , piece_id=MPDDCNRUUZMRYBEXCGFKOGJHAXXJXFRPKQNIE7AUK6OSEP3WDAQA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.770996430 (81h12m42.468564737s) [noise=true , address=145.79.227.32:4836 , piece_id=LTIOGO4JLKGYSZ4ZEF4RR2MM3JPYBOROG537MXTUMMU5LD6LOHTA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.771564952 (81h12m42.469133258s) [noise=true , address=145.79.227.20:4438 , piece_id=WXPDD7Q7FIRZL6XD4C5V6G5UTL4K5D2NBRDVGBWELCYRGELVJNIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772355418 (81h12m42.469923725s) [noise=true , address=145.79.227.39:4015 , piece_id=UTKFDAUOSNLSDIIEWSIVUIVVIQ7LVZD2BVVBEAG52S5X2UTHMVLQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772720121 (81h12m42.470288426s) [noise=true , address=145.79.227.34:4906 , piece_id=JSE5ZBJTJDC4ATEN36BP4S3TRS64OQX3YIZFEEN2EVB23JVJ5QUQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.772728877 (81h12m42.470297183s) [noise=true , address=145.79.227.20:4437 , piece_id=4SLNFSLHALWUU4LSELTRV3TJQ6N6EZBO3FDOOKPLEMAKGSNC6K6Q, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773353122 (81h12m42.470921429s) [noise=true , address=145.79.227.43:4158 , piece_id=LRMYWFEOLEDFNODKIHHQP6HDXPJMFSYD47LKWS6UA3PLIYNVS2GA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.773928217 (81h12m42.471496522s) [noise=true , address=145.79.227.33:4869 , piece_id=7VXDZ3KG4P4Z5ZOEPTECUC3EGZMHEZIBG2RLENV3FE3QZKPQUCIQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.774308768 (81h12m42.471877073s) [noise=true , address=145.79.227.22:4510 , piece_id=DUBIKMD2ZWQE3OS4A2VHZ2G5MI7GMAGGNFOQUHKU2UNRHUMPQMGA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.777437452 (81h12m42.475005768s) [noise=true , address=145.79.227.21:4473 , piece_id=GJFPZXBIINXKBIAAHJZNLU67VQAIO7W5QFRQXL5P4ASYBCTIPPBA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.780306472 (81h12m42.477874778s) [noise=true , address=145.79.227.32:4834 , piece_id=HYV7C7TQKP4F4ZQKSY7WVUMGV3SJMMDYHVAIYFZEGRRF2YHGVMYQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:41:54.783206629 (81h12m42.480774925s) [noise=true , address=145.79.227.26:4653 , piece_id=7HRZZI34QXWHDXMA67PRSG37BBRMXJ7G7VR5A3PLBX3AZKTAV7TQ, seg_pos={0 0}, upload=71278]: Done uploading piece. err: <nil> Dec 9 00:42:24.611357608 (81h13m12.308925954s) [noise=true , address=77.90.31.12:4592 , piece_id=UV23TFTJZOO3EM56IPX3GLAXPH4DKJONC6UFMCH5GC4TWRCVOVHA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611392572 (81h13m12.308960909s) [noise=true , address=77.90.31.15:4698 , piece_id=AMI7LXWHC7QZLDEHLV5AMUTT6JW3MDU6HRJKH44H2XGXNKJBDIEA, seg_pos={0 0}, upload=71278]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 00:42:24.611436074 (81h13m12.30900441s) [seg_pos={0 0}, upload=71278]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 00:42:24.629340400 (81h13m12.326908736s) [upload=71278]: Done uploading segments... Dec 9 00:42:24.629373512 (81h13m12.326941858s) [upload=71278]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 382.699387ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 13 out of 60 is failed (threshold=3) [X...X...X.........X........X...XX.....XX..X...X.....X.X.....] duration: 9.353541856s trace: https://jaeger.dev.storj.io/trace/73cbc41c8163da51 FAIL (2025-12-09 01:01:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =89.149.201.147:4019 , piece_id=44XAWC4PX6JDYDNQES7RFJOL44BKSMZL6JXD4VSNSJ63EFCU6IGA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.653731184 (81h27m19.35129949s) [noise=true , address=89.149.201.147:4020 , piece_id=6UT7ISSBNWIIRYDAJ7AFVXYNPNB2AJD36BWEX5UBM5ZUXYNZZBOA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.654018120 (81h27m19.351586436s) [noise=true , address=81.171.14.70:4047 , piece_id=Y3IFJKBSGE47TODR4FTRYSCQFPXYDAQK6PIPO4ARTIC7CLR56S4Q, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.654327128 (81h27m19.351895434s) [noise=true , address=185.167.204.205:4032 , piece_id=ZRKSY5TYZ47BN5NJD734DAXN544U6TDJJNSYDVU7KPHEF2BTJAMA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.654782739 (81h27m19.352351055s) [noise=true , address=185.167.204.205:4015 , piece_id=D5IU7DJK5QUUXDZSJST7LMPOTQ43LMPPRQ3P46VKORYA7QWHRB2Q, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.655140286 (81h27m19.352708602s) [noise=true , address=185.167.204.205:4035 , piece_id=EKHXDUJ6JUKWPGLRZI5QJGUBF4QQKJDBSKF64H7RNVS6Q3UXOSHA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.655534964 (81h27m19.35310328s) [noise=true , address=185.167.204.206:4066 , piece_id=KW6JXLGMA66TJ2X3BSQ53MYERCPKIBT76PPBVE3PFIUMUKIKPBIA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.655963444 (81h27m19.35353175s) [noise=true , address=89.149.201.147:4010 , piece_id=RKCPZSXPW6PL3CJZMOWMWWC6NRU3MRLZMUFX5ZPFHJI55U7ZIP7Q, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.656381366 (81h27m19.353949672s) [noise=true , address=185.167.204.205:4030 , piece_id=2WEVN7VFRXBRV277D5W5B42L3WDGYT2V4WSJTKSTMD3WBKFUUVNQ, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.656743802 (81h27m19.354312108s) [noise=true , address=89.149.201.147:4021 , piece_id=QXEVXQODAXU2YMITYDM7VYD2ZKM5CRLEOKN5VRBTOLRJ6ZIVEYXQ, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:31.657206457 (81h27m19.354774763s) [noise=true , address=89.149.201.147:4006 , piece_id=KGZQZGUW3HO7HT5R3NX4NSKZTSB7ND537INBKOGAKFQGRBNHJ3TA, seg_pos={0 0}, upload=71490]: Uploading piece... Dec 9 00:56:32.160763915 (81h27m19.858332231s) [upload=71490]: Next returned nil segment Dec 9 00:56:32.160772261 (81h27m19.858340566s) [upload=71490]: Waiting for error group managing segments... Dec 9 00:56:32.367091113 (81h27m20.06465945s) [noise=true , address=185.167.204.205:4013 , piece_id=NNBGYPP6DGVX3FC4DZMC43ZAMHFPSZMY7KGRRIIGLFKCM3SQHHPA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.382359607 (81h27m20.079927943s) [noise=true , address=185.167.204.205:4009 , piece_id=QAL7FK7ZORITTNCJT4MNNFNPKNBQ54DXRCH2J6X6E4OEYOMAG5AQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.401869895 (81h27m20.099438232s) [noise=true , address=89.149.201.147:4004 , piece_id=63FVBOL7WODTHOLQFC75CWPSUQCJWS7LODV5FBRH2SICL3K3XWCQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.406066094 (81h27m20.103634431s) [noise=true , address=89.149.201.147:4021 , piece_id=QXEVXQODAXU2YMITYDM7VYD2ZKM5CRLEOKN5VRBTOLRJ6ZIVEYXQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.411331421 (81h27m20.108899797s) [noise=true , address=185.167.204.205:4015 , piece_id=D5IU7DJK5QUUXDZSJST7LMPOTQ43LMPPRQ3P46VKORYA7QWHRB2Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.418812968 (81h27m20.116381303s) [noise=true , address=89.149.201.147:4005 , piece_id=BYT4SLGEUH2AKLMHKBZ6QSQVJKOY2RVWQJC56BRV62QEKTBWKFRA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.420572304 (81h27m20.118140651s) [noise=true , address=81.171.14.70:4047 , piece_id=Y3IFJKBSGE47TODR4FTRYSCQFPXYDAQK6PIPO4ARTIC7CLR56S4Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.527718158 (81h27m20.225286494s) [noise=true , address=81.171.14.70:4044 , piece_id=PKZTY4JWVBCHCUHLL5MXXVWQ5MWWMUWR7I6SVQNI7BTR27IDPRRQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.574245548 (81h27m20.271813885s) [noise=true , address=185.167.204.206:4066 , piece_id=KW6JXLGMA66TJ2X3BSQ53MYERCPKIBT76PPBVE3PFIUMUKIKPBIA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.811440733 (81h27m20.509009049s) [noise=true , address=89.149.201.147:4015 , piece_id=EYC7EULWHAVARKFIMAQEDCMWHPTUSJBDFCYJT4RVSLSLYA4LV2CA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.814580629 (81h27m20.512148934s) [noise=true , address=89.149.201.147:4006 , piece_id=KGZQZGUW3HO7HT5R3NX4NSKZTSB7ND537INBKOGAKFQGRBNHJ3TA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.842364613 (81h27m20.539932918s) [noise=true , address=89.149.201.147:4017 , piece_id=H4BUDJVTVHDT7KYBJCBYXVHSHW6XVUICOHPAWFDWUCAFDRSLGX3Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.873751909 (81h27m20.571320235s) [noise=true , address=89.149.201.147:4019 , piece_id=44XAWC4PX6JDYDNQES7RFJOL44BKSMZL6JXD4VSNSJ63EFCU6IGA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.887103492 (81h27m20.584671828s) [noise=true , address=81.171.14.70:4025 , piece_id=AHLEQXU6UBZR2AANMXY72UGDKKCNW5RTDGD7MMQO4W2MMHUPQLUA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.908520581 (81h27m20.606088898s) [noise=true , address=185.167.204.205:4026 , piece_id=2W4DN6TY7AJJWQXYWSS6YZIGN563IXIQ2RMYWG3XHEL47PC7CJYQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:32.952690147 (81h27m20.650258454s) [noise=true , address=185.167.204.206:4049 , piece_id=WJOGRXIB3TD7JRCE7EZA6Q2JYL3SVHWKCIGBQ7JPBXSUFEOK4I3A, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.008828632 (81h27m20.706396977s) [noise=true , address=81.171.14.70:4038 , piece_id=BACFOGLKUZOOCDGQJIYHQVCDTUOGFGNAEEDZK246ULZSNFIYEOMA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.103114172 (81h27m20.800682508s) [noise=true , address=185.167.204.206:4070 , piece_id=ELEURNR5CXP7624QVG2M2ZK7HHZM2GHQXL5CPYSA2ODQAOXE3NKA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.134718753 (81h27m20.832287079s) [noise=true , address=81.171.14.70:4045 , piece_id=JYLMRVFZJQQMYNJVJ2GFM3QABGZZIPZCGNO4JVKOUISUSF2ZGGZQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.208666869 (81h27m20.906235185s) [noise=true , address=89.149.201.147:4020 , piece_id=6UT7ISSBNWIIRYDAJ7AFVXYNPNB2AJD36BWEX5UBM5ZUXYNZZBOA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.502522334 (81h27m21.20009065s) [noise=true , address=81.171.14.70:4040 , piece_id=LS33AXEKVAVNEVBJZ4XVVNIVW54Y3GCPI3RY4QPA5JNLJ32PBS5Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.520716964 (81h27m21.2182853s) [noise=true , address=89.149.201.147:4010 , piece_id=RKCPZSXPW6PL3CJZMOWMWWC6NRU3MRLZMUFX5ZPFHJI55U7ZIP7Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.541974847 (81h27m21.239543173s) [noise=true , address=185.167.204.205:4035 , piece_id=EKHXDUJ6JUKWPGLRZI5QJGUBF4QQKJDBSKF64H7RNVS6Q3UXOSHA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.901468573 (81h27m21.599036879s) [noise=true , address=185.167.204.205:4030 , piece_id=2WEVN7VFRXBRV277D5W5B42L3WDGYT2V4WSJTKSTMD3WBKFUUVNQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:33.908515979 (81h27m21.606084295s) [noise=true , address=185.167.204.205:4032 , piece_id=ZRKSY5TYZ47BN5NJD734DAXN544U6TDJJNSYDVU7KPHEF2BTJAMA, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:34.121817866 (81h27m21.819386183s) [noise=true , address=81.171.14.70:4048 , piece_id=7W4TKSAVVTAZTOWVSI4PMDF57WOARYODCQOOYDBMTEUBDK26FM5A, seg_pos={0 0}, upload=71490]: Done uploading piece. err: <nil> Dec 9 00:56:34.121835138 (81h27m21.819403454s) [seg_pos={0 0}, upload=71490]: Segment reached optimal threshold of 26 pieces. Dec 9 00:56:34.122000397 (81h27m21.819568713s) [noise=true , address=89.149.201.147:4007 , piece_id=O52IHBKR3GBNALB7E2OCMVZJ6A4CXVNEPUDUQANAF2VATGZ5U46Q, seg_pos={0 0}, upload=71490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 00:56:34.122006168 (81h27m21.819574484s) [noise=true , address=89.149.201.147:4007 , piece_id=O52IHBKR3GBNALB7E2OCMVZJ6A4CXVNEPUDUQANAF2VATGZ5U46Q, seg_pos={0 0}, upload=71490]: Long tail cancellation due to optimal threshold Dec 9 00:56:34.122138234 (81h27m21.81970655s) [noise=true , address=81.171.14.70:4046 , piece_id=Q27COUPBROIQC4IVWVO55AJLV7BKYZACQI57NIDFOYNWLV5NITJQ, seg_pos={0 0}, upload=71490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: expected piece hash; context canceled; EOF Dec 9 00:56:34.122166547 (81h27m21.819734864s) [noise=true , address=81.171.14.70:4046 , piece_id=Q27COUPBROIQC4IVWVO55AJLV7BKYZACQI57NIDFOYNWLV5NITJQ, seg_pos={0 0}, upload=71490]: Long tail cancellation due to optimal threshold Dec 9 00:56:34.122190121 (81h27m21.819758438s) [seg_pos={0 0}, upload=71490]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 00:56:34.122256786 (81h27m21.819825102s) [upload=71490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 00:56:34.122261705 (81h27m21.819830011s) [upload=71490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 00:56:34.272548935 (81h27m21.970117241s) [upload=71490]: Done uploading segments... Dec 9 00:56:34.272567188 (81h27m21.970135495s) [upload=71490]: Upload finished. err: <nil> Duration: 9.353562585s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)