Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XX.XX.................................................] duration: 400.677618ms FAIL (2025-08-17 13:44:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11h39m18.963301239s) [upload=220201]: Got next segment. Inline: false Aug 17 13:05:38.232064688 (211h39m18.963332307s) [upload=220201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:38.232068765 (211h39m18.963336385s) [upload=220201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:38.260394616 (211h39m18.991662237s) [seg_pos={0 0}, upload=220201]: Begin upload segment... Aug 17 13:05:38.260422680 (211h39m18.99169031s) [seg_pos={0 0}, upload=220201]: Done begin upload segment. Aug 17 13:05:38.260441515 (211h39m18.991709135s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260457946 (211h39m18.991725576s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260717223 (211h39m18.991984853s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260731841 (211h39m18.991999471s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260959870 (211h39m18.9922275s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260968356 (211h39m18.992235986s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261198057 (211h39m18.992465677s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261233694 (211h39m18.992501334s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261415967 (211h39m18.992683597s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261464098 (211h39m18.992731728s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261631563 (211h39m18.992899193s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261670385 (211h39m18.992938005s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261838482 (211h39m18.993106102s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261906510 (211h39m18.99317414s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262041383 (211h39m18.993309003s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262119430 (211h39m18.993387071s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262267067 (211h39m18.993534687s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262472443 (211h39m18.993740064s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.335674416 (211h39m19.066942037s) [upload=220201]: Next returned nil segment Aug 17 13:05:38.335685137 (211h39m19.066952757s) [upload=220201]: Waiting for error group managing segments... Aug 17 13:05:38.343418127 (211h39m19.074685747s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.343638923 (211h39m19.074906542s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.375313802 (211h39m19.106581432s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.384478504 (211h39m19.115746134s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.394391995 (211h39m19.125659625s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.395010046 (211h39m19.126277676s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.402387698 (211h39m19.133655328s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.409234503 (211h39m19.140502123s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.428228938 (211h39m19.159496568s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.546929112 (211h39m19.278196732s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.562922516 (211h39m19.294190176s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.786308181 (211h39m19.517575811s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.858771665 (211h39m19.590039295s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:06:38.221240067 (211h40m18.952507717s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221311331 (211h40m18.952578981s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:104.36.82.58:4285): protocol: write tcp 5.161.89.252:35328->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221354321 (211h40m18.952621971s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: rpc: dial tcp 104.36.82.2:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221450012 (211h40m18.952717662s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:104.36.82.60:4343): protocol: rpc: dial tcp 104.36.82.60:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221534150 (211h40m18.952801799s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: write tcp 5.161.89.252:49358->104.36.82.18:4376: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221585135 (211h40m18.952852785s) [seg_pos={0 0}, upload=220201]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 13:06:38.236359888 (211h40m18.967627539s) [upload=220201]: Done uploading segments... Aug 17 13:06:38.236387120 (211h40m18.967654769s) [upload=220201]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 400.697255ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......XX.XX................................................] duration: 513.447284ms FAIL (2025-08-17 13:43:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11h39m18.963301239s) [upload=220201]: Got next segment. Inline: false Aug 17 13:05:38.232064688 (211h39m18.963332307s) [upload=220201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:38.232068765 (211h39m18.963336385s) [upload=220201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:38.260394616 (211h39m18.991662237s) [seg_pos={0 0}, upload=220201]: Begin upload segment... Aug 17 13:05:38.260422680 (211h39m18.99169031s) [seg_pos={0 0}, upload=220201]: Done begin upload segment. Aug 17 13:05:38.260441515 (211h39m18.991709135s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260457946 (211h39m18.991725576s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260717223 (211h39m18.991984853s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260731841 (211h39m18.991999471s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260959870 (211h39m18.9922275s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260968356 (211h39m18.992235986s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261198057 (211h39m18.992465677s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261233694 (211h39m18.992501334s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261415967 (211h39m18.992683597s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261464098 (211h39m18.992731728s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261631563 (211h39m18.992899193s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261670385 (211h39m18.992938005s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261838482 (211h39m18.993106102s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261906510 (211h39m18.99317414s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262041383 (211h39m18.993309003s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262119430 (211h39m18.993387071s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262267067 (211h39m18.993534687s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262472443 (211h39m18.993740064s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.335674416 (211h39m19.066942037s) [upload=220201]: Next returned nil segment Aug 17 13:05:38.335685137 (211h39m19.066952757s) [upload=220201]: Waiting for error group managing segments... Aug 17 13:05:38.343418127 (211h39m19.074685747s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.343638923 (211h39m19.074906542s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.375313802 (211h39m19.106581432s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.384478504 (211h39m19.115746134s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.394391995 (211h39m19.125659625s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.395010046 (211h39m19.126277676s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.402387698 (211h39m19.133655328s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.409234503 (211h39m19.140502123s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.428228938 (211h39m19.159496568s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.546929112 (211h39m19.278196732s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.562922516 (211h39m19.294190176s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.786308181 (211h39m19.517575811s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.858771665 (211h39m19.590039295s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:06:38.221240067 (211h40m18.952507717s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221311331 (211h40m18.952578981s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:104.36.82.58:4285): protocol: write tcp 5.161.89.252:35328->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221354321 (211h40m18.952621971s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: rpc: dial tcp 104.36.82.2:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221450012 (211h40m18.952717662s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:104.36.82.60:4343): protocol: rpc: dial tcp 104.36.82.60:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221534150 (211h40m18.952801799s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: write tcp 5.161.89.252:49358->104.36.82.18:4376: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221585135 (211h40m18.952852785s) [seg_pos={0 0}, upload=220201]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 13:06:38.236359888 (211h40m18.967627539s) [upload=220201]: Done uploading segments... Aug 17 13:06:38.236387120 (211h40m18.967654769s) [upload=220201]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 513.467183ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XXXXXX....................................................] duration: 451.87485ms FAIL (2025-08-17 13:43:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 451.898666ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 604.843946ms FAIL (2025-08-17 13:43:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 13:05:03.230244168 (211h38m43.961511788s) [upload=220194]: Starting upload Aug 17 13:05:03.230282931 (211h38m43.961550561s) [upload=220194]: Done starting upload Aug 17 13:05:03.230337775 (211h38m43.961605395s) [upload=220194]: Uploading segments... Aug 17 13:05:03.230498627 (211h38m43.961766257s) [upload=220194]: Got next segment. Inline: false Aug 17 13:05:03.230523233 (211h38m43.961790852s) [upload=220194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:03.230526769 (211h38m43.96179439s) [upload=220194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:03.253244145 (211h38m43.984511775s) [seg_pos={0 0}, upload=220194]: Begin upload segment... Aug 17 13:05:03.253284361 (211h38m43.984551981s) [seg_pos={0 0}, upload=220194]: Done begin upload segment. Aug 17 13:05:03.253314057 (211h38m43.984581687s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253326811 (211h38m43.984594431s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253379720 (211h38m43.98464735s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253678482 (211h38m43.984946102s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253743043 (211h38m43.985010673s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253961614 (211h38m43.985229234s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254079065 (211h38m43.985346696s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254257481 (211h38m43.985525111s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254408585 (211h38m43.985676204s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254523081 (211h38m43.985790721s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254695415 (211h38m43.985963045s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254849924 (211h38m43.986117554s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254965993 (211h38m43.986233613s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255073975 (211h38m43.986341606s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255232403 (211h38m43.986500033s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255467977 (211h38m43.986735607s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255719861 (211h38m43.986987481s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255961866 (211h38m43.987229496s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.323589897 (211h38m44.054857527s) [upload=220194]: Next returned nil segment Aug 17 13:05:03.323600959 (211h38m44.054868579s) [upload=220194]: Waiting for error group managing segments... Aug 17 13:05:03.351486953 (211h38m44.082754583s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.356136245 (211h38m44.087403865s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.359798501 (211h38m44.091066121s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.368712994 (211h38m44.099980613s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.451425619 (211h38m44.18269327s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.488338257 (211h38m44.219605887s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.533602960 (211h38m44.26487062s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.706703573 (211h38m44.437971194s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.845738394 (211h38m44.577006013s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.854673865 (211h38m44.585941515s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:06:03.213670588 (211h39m43.944938208s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213738616 (211h39m43.945006266s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 5.161.89.252:44086->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213807617 (211h39m43.945075236s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213844345 (211h39m43.945111965s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213873840 (211h39m43.945141471s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:16954->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213895070 (211h39m43.945162691s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213912293 (211h39m43.945179912s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213916270 (211h39m43.94518389s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213938842 (211h39m43.945206472s) [seg_pos={0 0}, upload=220194]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 13:06:03.239464369 (211h39m43.970731999s) [upload=220194]: Done uploading segments... Aug 17 13:06:03.239484097 (211h39m43.970751716s) [upload=220194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 604.868783ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XX.XX...............................................] duration: 666.486232ms FAIL (2025-08-17 13:43:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11h39m18.963301239s) [upload=220201]: Got next segment. Inline: false Aug 17 13:05:38.232064688 (211h39m18.963332307s) [upload=220201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:38.232068765 (211h39m18.963336385s) [upload=220201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:38.260394616 (211h39m18.991662237s) [seg_pos={0 0}, upload=220201]: Begin upload segment... Aug 17 13:05:38.260422680 (211h39m18.99169031s) [seg_pos={0 0}, upload=220201]: Done begin upload segment. Aug 17 13:05:38.260441515 (211h39m18.991709135s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260457946 (211h39m18.991725576s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260717223 (211h39m18.991984853s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260731841 (211h39m18.991999471s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260959870 (211h39m18.9922275s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260968356 (211h39m18.992235986s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261198057 (211h39m18.992465677s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261233694 (211h39m18.992501334s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261415967 (211h39m18.992683597s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261464098 (211h39m18.992731728s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261631563 (211h39m18.992899193s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261670385 (211h39m18.992938005s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261838482 (211h39m18.993106102s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261906510 (211h39m18.99317414s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262041383 (211h39m18.993309003s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262119430 (211h39m18.993387071s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262267067 (211h39m18.993534687s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262472443 (211h39m18.993740064s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.335674416 (211h39m19.066942037s) [upload=220201]: Next returned nil segment Aug 17 13:05:38.335685137 (211h39m19.066952757s) [upload=220201]: Waiting for error group managing segments... Aug 17 13:05:38.343418127 (211h39m19.074685747s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.343638923 (211h39m19.074906542s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.375313802 (211h39m19.106581432s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.384478504 (211h39m19.115746134s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.394391995 (211h39m19.125659625s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.395010046 (211h39m19.126277676s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.402387698 (211h39m19.133655328s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.409234503 (211h39m19.140502123s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.428228938 (211h39m19.159496568s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.546929112 (211h39m19.278196732s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.562922516 (211h39m19.294190176s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.786308181 (211h39m19.517575811s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.858771665 (211h39m19.590039295s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:06:38.221240067 (211h40m18.952507717s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221311331 (211h40m18.952578981s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:104.36.82.58:4285): protocol: write tcp 5.161.89.252:35328->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221354321 (211h40m18.952621971s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: rpc: dial tcp 104.36.82.2:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221450012 (211h40m18.952717662s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:104.36.82.60:4343): protocol: rpc: dial tcp 104.36.82.60:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221534150 (211h40m18.952801799s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: write tcp 5.161.89.252:49358->104.36.82.18:4376: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221585135 (211h40m18.952852785s) [seg_pos={0 0}, upload=220201]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 13:06:38.236359888 (211h40m18.967627539s) [upload=220201]: Done uploading segments... Aug 17 13:06:38.236387120 (211h40m18.967654769s) [upload=220201]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 666.515377ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XXXXXX...................................................] duration: 592.515531ms FAIL (2025-08-17 13:43:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 592.529317ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XXXX.......................................................] duration: 759.1575ms FAIL (2025-08-17 13:43:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 13:05:03.230244168 (211h38m43.961511788s) [upload=220194]: Starting upload Aug 17 13:05:03.230282931 (211h38m43.961550561s) [upload=220194]: Done starting upload Aug 17 13:05:03.230337775 (211h38m43.961605395s) [upload=220194]: Uploading segments... Aug 17 13:05:03.230498627 (211h38m43.961766257s) [upload=220194]: Got next segment. Inline: false Aug 17 13:05:03.230523233 (211h38m43.961790852s) [upload=220194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:03.230526769 (211h38m43.96179439s) [upload=220194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:03.253244145 (211h38m43.984511775s) [seg_pos={0 0}, upload=220194]: Begin upload segment... Aug 17 13:05:03.253284361 (211h38m43.984551981s) [seg_pos={0 0}, upload=220194]: Done begin upload segment. Aug 17 13:05:03.253314057 (211h38m43.984581687s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253326811 (211h38m43.984594431s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253379720 (211h38m43.98464735s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253678482 (211h38m43.984946102s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253743043 (211h38m43.985010673s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253961614 (211h38m43.985229234s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254079065 (211h38m43.985346696s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254257481 (211h38m43.985525111s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254408585 (211h38m43.985676204s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254523081 (211h38m43.985790721s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254695415 (211h38m43.985963045s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254849924 (211h38m43.986117554s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254965993 (211h38m43.986233613s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255073975 (211h38m43.986341606s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255232403 (211h38m43.986500033s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255467977 (211h38m43.986735607s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255719861 (211h38m43.986987481s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255961866 (211h38m43.987229496s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.323589897 (211h38m44.054857527s) [upload=220194]: Next returned nil segment Aug 17 13:05:03.323600959 (211h38m44.054868579s) [upload=220194]: Waiting for error group managing segments... Aug 17 13:05:03.351486953 (211h38m44.082754583s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.356136245 (211h38m44.087403865s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.359798501 (211h38m44.091066121s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.368712994 (211h38m44.099980613s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.451425619 (211h38m44.18269327s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.488338257 (211h38m44.219605887s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.533602960 (211h38m44.26487062s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.706703573 (211h38m44.437971194s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.845738394 (211h38m44.577006013s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.854673865 (211h38m44.585941515s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:06:03.213670588 (211h39m43.944938208s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213738616 (211h39m43.945006266s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 5.161.89.252:44086->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213807617 (211h39m43.945075236s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213844345 (211h39m43.945111965s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213873840 (211h39m43.945141471s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:16954->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213895070 (211h39m43.945162691s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213912293 (211h39m43.945179912s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213916270 (211h39m43.94518389s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213938842 (211h39m43.945206472s) [seg_pos={0 0}, upload=220194]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 13:06:03.239464369 (211h39m43.970731999s) [upload=220194]: Done uploading segments... Aug 17 13:06:03.239484097 (211h39m43.970751716s) [upload=220194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 759.17333ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XX.XX..............................................] duration: 544.778507ms FAIL (2025-08-17 13:42:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11h39m18.963301239s) [upload=220201]: Got next segment. Inline: false Aug 17 13:05:38.232064688 (211h39m18.963332307s) [upload=220201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:38.232068765 (211h39m18.963336385s) [upload=220201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:38.260394616 (211h39m18.991662237s) [seg_pos={0 0}, upload=220201]: Begin upload segment... Aug 17 13:05:38.260422680 (211h39m18.99169031s) [seg_pos={0 0}, upload=220201]: Done begin upload segment. Aug 17 13:05:38.260441515 (211h39m18.991709135s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260457946 (211h39m18.991725576s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260717223 (211h39m18.991984853s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260731841 (211h39m18.991999471s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260959870 (211h39m18.9922275s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260968356 (211h39m18.992235986s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261198057 (211h39m18.992465677s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261233694 (211h39m18.992501334s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261415967 (211h39m18.992683597s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261464098 (211h39m18.992731728s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261631563 (211h39m18.992899193s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261670385 (211h39m18.992938005s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261838482 (211h39m18.993106102s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261906510 (211h39m18.99317414s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262041383 (211h39m18.993309003s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262119430 (211h39m18.993387071s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262267067 (211h39m18.993534687s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262472443 (211h39m18.993740064s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.335674416 (211h39m19.066942037s) [upload=220201]: Next returned nil segment Aug 17 13:05:38.335685137 (211h39m19.066952757s) [upload=220201]: Waiting for error group managing segments... Aug 17 13:05:38.343418127 (211h39m19.074685747s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.343638923 (211h39m19.074906542s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.375313802 (211h39m19.106581432s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.384478504 (211h39m19.115746134s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.394391995 (211h39m19.125659625s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.395010046 (211h39m19.126277676s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.402387698 (211h39m19.133655328s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.409234503 (211h39m19.140502123s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.428228938 (211h39m19.159496568s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.546929112 (211h39m19.278196732s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.562922516 (211h39m19.294190176s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.786308181 (211h39m19.517575811s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.858771665 (211h39m19.590039295s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:06:38.221240067 (211h40m18.952507717s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221311331 (211h40m18.952578981s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:104.36.82.58:4285): protocol: write tcp 5.161.89.252:35328->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221354321 (211h40m18.952621971s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: rpc: dial tcp 104.36.82.2:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221450012 (211h40m18.952717662s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:104.36.82.60:4343): protocol: rpc: dial tcp 104.36.82.60:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221534150 (211h40m18.952801799s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: write tcp 5.161.89.252:49358->104.36.82.18:4376: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221585135 (211h40m18.952852785s) [seg_pos={0 0}, upload=220201]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 13:06:38.236359888 (211h40m18.967627539s) [upload=220201]: Done uploading segments... Aug 17 13:06:38.236387120 (211h40m18.967654769s) [upload=220201]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 544.802421ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XXXXXX..................................................] duration: 427.272065ms FAIL (2025-08-17 13:42:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 427.302452ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 179.481169ms FAIL (2025-08-17 13:42:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:00:24.431681640 (211h34m5.162949261s) [upload=220113]: Starting upload Aug 17 13:00:24.431716026 (211h34m5.162983646s) [upload=220113]: Done starting upload Aug 17 13:00:24.431759107 (211h34m5.163026727s) [upload=220113]: Uploading segments... Aug 17 13:00:24.431790736 (211h34m5.163058366s) [upload=220113]: Got next segment. Inline: true Aug 17 13:00:24.431795495 (211h34m5.163063115s) [upload=220113]: Waiting for error group managing segments... Aug 17 13:00:24.431947481 (211h34m5.163215111s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:00:24.431953533 (211h34m5.163221163s) [upload=220113]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:00:24.431956157 (211h34m5.163223787s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:00:24.434700137 (211h34m5.165967787s) [upload=220113]: Done uploading segments... Aug 17 13:00:24.434709414 (211h34m5.165977034s) [upload=220113]: Upload finished. err: metaclient: Too Many Requests Duration:* 179.507398ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XXXX......................................................] duration: 533.175667ms FAIL (2025-08-17 13:42:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 13:05:03.230244168 (211h38m43.961511788s) [upload=220194]: Starting upload Aug 17 13:05:03.230282931 (211h38m43.961550561s) [upload=220194]: Done starting upload Aug 17 13:05:03.230337775 (211h38m43.961605395s) [upload=220194]: Uploading segments... Aug 17 13:05:03.230498627 (211h38m43.961766257s) [upload=220194]: Got next segment. Inline: false Aug 17 13:05:03.230523233 (211h38m43.961790852s) [upload=220194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:03.230526769 (211h38m43.96179439s) [upload=220194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:03.253244145 (211h38m43.984511775s) [seg_pos={0 0}, upload=220194]: Begin upload segment... Aug 17 13:05:03.253284361 (211h38m43.984551981s) [seg_pos={0 0}, upload=220194]: Done begin upload segment. Aug 17 13:05:03.253314057 (211h38m43.984581687s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253326811 (211h38m43.984594431s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253379720 (211h38m43.98464735s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253678482 (211h38m43.984946102s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253743043 (211h38m43.985010673s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253961614 (211h38m43.985229234s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254079065 (211h38m43.985346696s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254257481 (211h38m43.985525111s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254408585 (211h38m43.985676204s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254523081 (211h38m43.985790721s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254695415 (211h38m43.985963045s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254849924 (211h38m43.986117554s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254965993 (211h38m43.986233613s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255073975 (211h38m43.986341606s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255232403 (211h38m43.986500033s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255467977 (211h38m43.986735607s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255719861 (211h38m43.986987481s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255961866 (211h38m43.987229496s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.323589897 (211h38m44.054857527s) [upload=220194]: Next returned nil segment Aug 17 13:05:03.323600959 (211h38m44.054868579s) [upload=220194]: Waiting for error group managing segments... Aug 17 13:05:03.351486953 (211h38m44.082754583s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.356136245 (211h38m44.087403865s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.359798501 (211h38m44.091066121s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.368712994 (211h38m44.099980613s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.451425619 (211h38m44.18269327s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.488338257 (211h38m44.219605887s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.533602960 (211h38m44.26487062s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.706703573 (211h38m44.437971194s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.845738394 (211h38m44.577006013s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.854673865 (211h38m44.585941515s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:06:03.213670588 (211h39m43.944938208s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213738616 (211h39m43.945006266s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 5.161.89.252:44086->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213807617 (211h39m43.945075236s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213844345 (211h39m43.945111965s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213873840 (211h39m43.945141471s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:16954->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213895070 (211h39m43.945162691s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213912293 (211h39m43.945179912s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213916270 (211h39m43.94518389s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213938842 (211h39m43.945206472s) [seg_pos={0 0}, upload=220194]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 13:06:03.239464369 (211h39m43.970731999s) [upload=220194]: Done uploading segments... Aug 17 13:06:03.239484097 (211h39m43.970751716s) [upload=220194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 533.198951ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XX.XX.............................................] duration: 530.793618ms FAIL (2025-08-17 13:41:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11h39m18.963301239s) [upload=220201]: Got next segment. Inline: false Aug 17 13:05:38.232064688 (211h39m18.963332307s) [upload=220201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:38.232068765 (211h39m18.963336385s) [upload=220201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:38.260394616 (211h39m18.991662237s) [seg_pos={0 0}, upload=220201]: Begin upload segment... Aug 17 13:05:38.260422680 (211h39m18.99169031s) [seg_pos={0 0}, upload=220201]: Done begin upload segment. Aug 17 13:05:38.260441515 (211h39m18.991709135s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260457946 (211h39m18.991725576s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260717223 (211h39m18.991984853s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260731841 (211h39m18.991999471s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260959870 (211h39m18.9922275s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260968356 (211h39m18.992235986s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261198057 (211h39m18.992465677s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261233694 (211h39m18.992501334s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261415967 (211h39m18.992683597s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261464098 (211h39m18.992731728s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261631563 (211h39m18.992899193s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261670385 (211h39m18.992938005s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261838482 (211h39m18.993106102s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261906510 (211h39m18.99317414s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262041383 (211h39m18.993309003s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262119430 (211h39m18.993387071s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262267067 (211h39m18.993534687s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262472443 (211h39m18.993740064s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.335674416 (211h39m19.066942037s) [upload=220201]: Next returned nil segment Aug 17 13:05:38.335685137 (211h39m19.066952757s) [upload=220201]: Waiting for error group managing segments... Aug 17 13:05:38.343418127 (211h39m19.074685747s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.343638923 (211h39m19.074906542s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.375313802 (211h39m19.106581432s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.384478504 (211h39m19.115746134s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.394391995 (211h39m19.125659625s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.395010046 (211h39m19.126277676s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.402387698 (211h39m19.133655328s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.409234503 (211h39m19.140502123s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.428228938 (211h39m19.159496568s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.546929112 (211h39m19.278196732s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.562922516 (211h39m19.294190176s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.786308181 (211h39m19.517575811s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.858771665 (211h39m19.590039295s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:06:38.221240067 (211h40m18.952507717s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221311331 (211h40m18.952578981s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:104.36.82.58:4285): protocol: write tcp 5.161.89.252:35328->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221354321 (211h40m18.952621971s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: rpc: dial tcp 104.36.82.2:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221450012 (211h40m18.952717662s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:104.36.82.60:4343): protocol: rpc: dial tcp 104.36.82.60:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221534150 (211h40m18.952801799s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: write tcp 5.161.89.252:49358->104.36.82.18:4376: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221585135 (211h40m18.952852785s) [seg_pos={0 0}, upload=220201]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 13:06:38.236359888 (211h40m18.967627539s) [upload=220201]: Done uploading segments... Aug 17 13:06:38.236387120 (211h40m18.967654769s) [upload=220201]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 530.820559ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XXXXXX.................................................] duration: 407.39729ms FAIL (2025-08-17 13:41:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 407.426563ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 229.660669ms FAIL (2025-08-17 13:41:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:00:24.431681640 (211h34m5.162949261s) [upload=220113]: Starting upload Aug 17 13:00:24.431716026 (211h34m5.162983646s) [upload=220113]: Done starting upload Aug 17 13:00:24.431759107 (211h34m5.163026727s) [upload=220113]: Uploading segments... Aug 17 13:00:24.431790736 (211h34m5.163058366s) [upload=220113]: Got next segment. Inline: true Aug 17 13:00:24.431795495 (211h34m5.163063115s) [upload=220113]: Waiting for error group managing segments... Aug 17 13:00:24.431947481 (211h34m5.163215111s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:00:24.431953533 (211h34m5.163221163s) [upload=220113]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:00:24.431956157 (211h34m5.163223787s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:00:24.434700137 (211h34m5.165967787s) [upload=220113]: Done uploading segments... Aug 17 13:00:24.434709414 (211h34m5.165977034s) [upload=220113]: Upload finished. err: metaclient: Too Many Requests Duration:* 229.679405ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XXXX.....................................................] duration: 459.839979ms FAIL (2025-08-17 13:40:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 13:05:03.230244168 (211h38m43.961511788s) [upload=220194]: Starting upload Aug 17 13:05:03.230282931 (211h38m43.961550561s) [upload=220194]: Done starting upload Aug 17 13:05:03.230337775 (211h38m43.961605395s) [upload=220194]: Uploading segments... Aug 17 13:05:03.230498627 (211h38m43.961766257s) [upload=220194]: Got next segment. Inline: false Aug 17 13:05:03.230523233 (211h38m43.961790852s) [upload=220194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:03.230526769 (211h38m43.96179439s) [upload=220194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:03.253244145 (211h38m43.984511775s) [seg_pos={0 0}, upload=220194]: Begin upload segment... Aug 17 13:05:03.253284361 (211h38m43.984551981s) [seg_pos={0 0}, upload=220194]: Done begin upload segment. Aug 17 13:05:03.253314057 (211h38m43.984581687s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253326811 (211h38m43.984594431s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253379720 (211h38m43.98464735s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253678482 (211h38m43.984946102s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253743043 (211h38m43.985010673s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253961614 (211h38m43.985229234s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254079065 (211h38m43.985346696s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254257481 (211h38m43.985525111s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254408585 (211h38m43.985676204s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254523081 (211h38m43.985790721s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254695415 (211h38m43.985963045s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254849924 (211h38m43.986117554s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254965993 (211h38m43.986233613s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255073975 (211h38m43.986341606s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255232403 (211h38m43.986500033s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255467977 (211h38m43.986735607s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255719861 (211h38m43.986987481s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255961866 (211h38m43.987229496s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.323589897 (211h38m44.054857527s) [upload=220194]: Next returned nil segment Aug 17 13:05:03.323600959 (211h38m44.054868579s) [upload=220194]: Waiting for error group managing segments... Aug 17 13:05:03.351486953 (211h38m44.082754583s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.356136245 (211h38m44.087403865s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.359798501 (211h38m44.091066121s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.368712994 (211h38m44.099980613s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.451425619 (211h38m44.18269327s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.488338257 (211h38m44.219605887s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.533602960 (211h38m44.26487062s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.706703573 (211h38m44.437971194s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.845738394 (211h38m44.577006013s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.854673865 (211h38m44.585941515s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:06:03.213670588 (211h39m43.944938208s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213738616 (211h39m43.945006266s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 5.161.89.252:44086->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213807617 (211h39m43.945075236s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213844345 (211h39m43.945111965s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213873840 (211h39m43.945141471s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:16954->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213895070 (211h39m43.945162691s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213912293 (211h39m43.945179912s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213916270 (211h39m43.94518389s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213938842 (211h39m43.945206472s) [seg_pos={0 0}, upload=220194]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 13:06:03.239464369 (211h39m43.970731999s) [upload=220194]: Done uploading segments... Aug 17 13:06:03.239484097 (211h39m43.970751716s) [upload=220194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 459.861228ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......XXXXXX................................................] duration: 324.906768ms FAIL (2025-08-17 13:40:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 324.926766ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........XX.XX............................................] duration: 614.396963ms FAIL (2025-08-17 13:40:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11h39m18.963301239s) [upload=220201]: Got next segment. Inline: false Aug 17 13:05:38.232064688 (211h39m18.963332307s) [upload=220201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:38.232068765 (211h39m18.963336385s) [upload=220201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:38.260394616 (211h39m18.991662237s) [seg_pos={0 0}, upload=220201]: Begin upload segment... Aug 17 13:05:38.260422680 (211h39m18.99169031s) [seg_pos={0 0}, upload=220201]: Done begin upload segment. Aug 17 13:05:38.260441515 (211h39m18.991709135s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260457946 (211h39m18.991725576s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260717223 (211h39m18.991984853s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260731841 (211h39m18.991999471s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260959870 (211h39m18.9922275s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260968356 (211h39m18.992235986s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261198057 (211h39m18.992465677s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261233694 (211h39m18.992501334s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261415967 (211h39m18.992683597s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261464098 (211h39m18.992731728s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261631563 (211h39m18.992899193s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261670385 (211h39m18.992938005s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261838482 (211h39m18.993106102s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261906510 (211h39m18.99317414s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262041383 (211h39m18.993309003s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262119430 (211h39m18.993387071s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262267067 (211h39m18.993534687s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262472443 (211h39m18.993740064s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.335674416 (211h39m19.066942037s) [upload=220201]: Next returned nil segment Aug 17 13:05:38.335685137 (211h39m19.066952757s) [upload=220201]: Waiting for error group managing segments... Aug 17 13:05:38.343418127 (211h39m19.074685747s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.343638923 (211h39m19.074906542s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.375313802 (211h39m19.106581432s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.384478504 (211h39m19.115746134s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.394391995 (211h39m19.125659625s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.395010046 (211h39m19.126277676s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.402387698 (211h39m19.133655328s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.409234503 (211h39m19.140502123s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.428228938 (211h39m19.159496568s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.546929112 (211h39m19.278196732s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.562922516 (211h39m19.294190176s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.786308181 (211h39m19.517575811s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.858771665 (211h39m19.590039295s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:06:38.221240067 (211h40m18.952507717s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221311331 (211h40m18.952578981s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:104.36.82.58:4285): protocol: write tcp 5.161.89.252:35328->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221354321 (211h40m18.952621971s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: rpc: dial tcp 104.36.82.2:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221450012 (211h40m18.952717662s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:104.36.82.60:4343): protocol: rpc: dial tcp 104.36.82.60:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221534150 (211h40m18.952801799s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: write tcp 5.161.89.252:49358->104.36.82.18:4376: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221585135 (211h40m18.952852785s) [seg_pos={0 0}, upload=220201]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 13:06:38.236359888 (211h40m18.967627539s) [upload=220201]: Done uploading segments... Aug 17 13:06:38.236387120 (211h40m18.967654769s) [upload=220201]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 614.438232ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 218.278142ms FAIL (2025-08-17 13:40:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:00:24.431681640 (211h34m5.162949261s) [upload=220113]: Starting upload Aug 17 13:00:24.431716026 (211h34m5.162983646s) [upload=220113]: Done starting upload Aug 17 13:00:24.431759107 (211h34m5.163026727s) [upload=220113]: Uploading segments... Aug 17 13:00:24.431790736 (211h34m5.163058366s) [upload=220113]: Got next segment. Inline: true Aug 17 13:00:24.431795495 (211h34m5.163063115s) [upload=220113]: Waiting for error group managing segments... Aug 17 13:00:24.431947481 (211h34m5.163215111s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:00:24.431953533 (211h34m5.163221163s) [upload=220113]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:00:24.431956157 (211h34m5.163223787s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:00:24.434700137 (211h34m5.165967787s) [upload=220113]: Done uploading segments... Aug 17 13:00:24.434709414 (211h34m5.165977034s) [upload=220113]: Upload finished. err: metaclient: Too Many Requests Duration:* 218.344116ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X...XXXX....................................................] duration: 918.071899ms FAIL (2025-08-17 13:40:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 13:05:03.230244168 (211h38m43.961511788s) [upload=220194]: Starting upload Aug 17 13:05:03.230282931 (211h38m43.961550561s) [upload=220194]: Done starting upload Aug 17 13:05:03.230337775 (211h38m43.961605395s) [upload=220194]: Uploading segments... Aug 17 13:05:03.230498627 (211h38m43.961766257s) [upload=220194]: Got next segment. Inline: false Aug 17 13:05:03.230523233 (211h38m43.961790852s) [upload=220194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:03.230526769 (211h38m43.96179439s) [upload=220194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:03.253244145 (211h38m43.984511775s) [seg_pos={0 0}, upload=220194]: Begin upload segment... Aug 17 13:05:03.253284361 (211h38m43.984551981s) [seg_pos={0 0}, upload=220194]: Done begin upload segment. Aug 17 13:05:03.253314057 (211h38m43.984581687s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253326811 (211h38m43.984594431s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253379720 (211h38m43.98464735s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253678482 (211h38m43.984946102s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253743043 (211h38m43.985010673s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253961614 (211h38m43.985229234s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254079065 (211h38m43.985346696s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254257481 (211h38m43.985525111s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254408585 (211h38m43.985676204s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254523081 (211h38m43.985790721s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254695415 (211h38m43.985963045s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254849924 (211h38m43.986117554s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254965993 (211h38m43.986233613s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255073975 (211h38m43.986341606s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255232403 (211h38m43.986500033s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255467977 (211h38m43.986735607s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255719861 (211h38m43.986987481s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255961866 (211h38m43.987229496s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.323589897 (211h38m44.054857527s) [upload=220194]: Next returned nil segment Aug 17 13:05:03.323600959 (211h38m44.054868579s) [upload=220194]: Waiting for error group managing segments... Aug 17 13:05:03.351486953 (211h38m44.082754583s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.356136245 (211h38m44.087403865s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.359798501 (211h38m44.091066121s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.368712994 (211h38m44.099980613s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.451425619 (211h38m44.18269327s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.488338257 (211h38m44.219605887s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.533602960 (211h38m44.26487062s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.706703573 (211h38m44.437971194s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.845738394 (211h38m44.577006013s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.854673865 (211h38m44.585941515s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:06:03.213670588 (211h39m43.944938208s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213738616 (211h39m43.945006266s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 5.161.89.252:44086->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213807617 (211h39m43.945075236s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213844345 (211h39m43.945111965s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213873840 (211h39m43.945141471s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:16954->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213895070 (211h39m43.945162691s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213912293 (211h39m43.945179912s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213916270 (211h39m43.94518389s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213938842 (211h39m43.945206472s) [seg_pos={0 0}, upload=220194]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 13:06:03.239464369 (211h39m43.970731999s) [upload=220194]: Done uploading segments... Aug 17 13:06:03.239484097 (211h39m43.970751716s) [upload=220194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 918.102216ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......XXXXXX...............................................] duration: 438.658618ms FAIL (2025-08-17 13:40:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 438.685349ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............XX.XX...........................................] duration: 559.505446ms FAIL (2025-08-17 13:40:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11h39m18.963301239s) [upload=220201]: Got next segment. Inline: false Aug 17 13:05:38.232064688 (211h39m18.963332307s) [upload=220201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:38.232068765 (211h39m18.963336385s) [upload=220201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:38.260394616 (211h39m18.991662237s) [seg_pos={0 0}, upload=220201]: Begin upload segment... Aug 17 13:05:38.260422680 (211h39m18.99169031s) [seg_pos={0 0}, upload=220201]: Done begin upload segment. Aug 17 13:05:38.260441515 (211h39m18.991709135s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260457946 (211h39m18.991725576s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260717223 (211h39m18.991984853s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260731841 (211h39m18.991999471s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260959870 (211h39m18.9922275s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260968356 (211h39m18.992235986s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261198057 (211h39m18.992465677s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261233694 (211h39m18.992501334s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261415967 (211h39m18.992683597s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261464098 (211h39m18.992731728s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261631563 (211h39m18.992899193s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261670385 (211h39m18.992938005s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261838482 (211h39m18.993106102s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261906510 (211h39m18.99317414s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262041383 (211h39m18.993309003s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262119430 (211h39m18.993387071s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262267067 (211h39m18.993534687s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262472443 (211h39m18.993740064s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.335674416 (211h39m19.066942037s) [upload=220201]: Next returned nil segment Aug 17 13:05:38.335685137 (211h39m19.066952757s) [upload=220201]: Waiting for error group managing segments... Aug 17 13:05:38.343418127 (211h39m19.074685747s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.343638923 (211h39m19.074906542s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.375313802 (211h39m19.106581432s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.384478504 (211h39m19.115746134s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.394391995 (211h39m19.125659625s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.395010046 (211h39m19.126277676s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.402387698 (211h39m19.133655328s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.409234503 (211h39m19.140502123s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.428228938 (211h39m19.159496568s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.546929112 (211h39m19.278196732s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.562922516 (211h39m19.294190176s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.786308181 (211h39m19.517575811s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.858771665 (211h39m19.590039295s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:06:38.221240067 (211h40m18.952507717s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221311331 (211h40m18.952578981s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:104.36.82.58:4285): protocol: write tcp 5.161.89.252:35328->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221354321 (211h40m18.952621971s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: rpc: dial tcp 104.36.82.2:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221450012 (211h40m18.952717662s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:104.36.82.60:4343): protocol: rpc: dial tcp 104.36.82.60:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221534150 (211h40m18.952801799s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: write tcp 5.161.89.252:49358->104.36.82.18:4376: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221585135 (211h40m18.952852785s) [seg_pos={0 0}, upload=220201]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 13:06:38.236359888 (211h40m18.967627539s) [upload=220201]: Done uploading segments... Aug 17 13:06:38.236387120 (211h40m18.967654769s) [upload=220201]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 559.531283ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........XXXXXX..............................................] duration: 541.471476ms FAIL (2025-08-17 13:39:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 541.502725ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X...XXXX...................................................] duration: 567.415307ms FAIL (2025-08-17 13:39:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 13:05:03.230244168 (211h38m43.961511788s) [upload=220194]: Starting upload Aug 17 13:05:03.230282931 (211h38m43.961550561s) [upload=220194]: Done starting upload Aug 17 13:05:03.230337775 (211h38m43.961605395s) [upload=220194]: Uploading segments... Aug 17 13:05:03.230498627 (211h38m43.961766257s) [upload=220194]: Got next segment. Inline: false Aug 17 13:05:03.230523233 (211h38m43.961790852s) [upload=220194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:03.230526769 (211h38m43.96179439s) [upload=220194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:03.253244145 (211h38m43.984511775s) [seg_pos={0 0}, upload=220194]: Begin upload segment... Aug 17 13:05:03.253284361 (211h38m43.984551981s) [seg_pos={0 0}, upload=220194]: Done begin upload segment. Aug 17 13:05:03.253314057 (211h38m43.984581687s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253326811 (211h38m43.984594431s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253379720 (211h38m43.98464735s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253678482 (211h38m43.984946102s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253743043 (211h38m43.985010673s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253961614 (211h38m43.985229234s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254079065 (211h38m43.985346696s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254257481 (211h38m43.985525111s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254408585 (211h38m43.985676204s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254523081 (211h38m43.985790721s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254695415 (211h38m43.985963045s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254849924 (211h38m43.986117554s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254965993 (211h38m43.986233613s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255073975 (211h38m43.986341606s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255232403 (211h38m43.986500033s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255467977 (211h38m43.986735607s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255719861 (211h38m43.986987481s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255961866 (211h38m43.987229496s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.323589897 (211h38m44.054857527s) [upload=220194]: Next returned nil segment Aug 17 13:05:03.323600959 (211h38m44.054868579s) [upload=220194]: Waiting for error group managing segments... Aug 17 13:05:03.351486953 (211h38m44.082754583s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.356136245 (211h38m44.087403865s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.359798501 (211h38m44.091066121s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.368712994 (211h38m44.099980613s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.451425619 (211h38m44.18269327s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.488338257 (211h38m44.219605887s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.533602960 (211h38m44.26487062s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.706703573 (211h38m44.437971194s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.845738394 (211h38m44.577006013s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.854673865 (211h38m44.585941515s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:06:03.213670588 (211h39m43.944938208s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213738616 (211h39m43.945006266s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 5.161.89.252:44086->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213807617 (211h39m43.945075236s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213844345 (211h39m43.945111965s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213873840 (211h39m43.945141471s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:16954->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213895070 (211h39m43.945162691s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213912293 (211h39m43.945179912s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213916270 (211h39m43.94518389s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213938842 (211h39m43.945206472s) [seg_pos={0 0}, upload=220194]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 13:06:03.239464369 (211h39m43.970731999s) [upload=220194]: Done uploading segments... Aug 17 13:06:03.239484097 (211h39m43.970751716s) [upload=220194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 567.445665ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............XX.XX..........................................] duration: 411.039446ms FAIL (2025-08-17 13:39:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11h39m18.963301239s) [upload=220201]: Got next segment. Inline: false Aug 17 13:05:38.232064688 (211h39m18.963332307s) [upload=220201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:38.232068765 (211h39m18.963336385s) [upload=220201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:38.260394616 (211h39m18.991662237s) [seg_pos={0 0}, upload=220201]: Begin upload segment... Aug 17 13:05:38.260422680 (211h39m18.99169031s) [seg_pos={0 0}, upload=220201]: Done begin upload segment. Aug 17 13:05:38.260441515 (211h39m18.991709135s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260457946 (211h39m18.991725576s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260717223 (211h39m18.991984853s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260731841 (211h39m18.991999471s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260959870 (211h39m18.9922275s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260968356 (211h39m18.992235986s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261198057 (211h39m18.992465677s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261233694 (211h39m18.992501334s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261415967 (211h39m18.992683597s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261464098 (211h39m18.992731728s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261631563 (211h39m18.992899193s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261670385 (211h39m18.992938005s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261838482 (211h39m18.993106102s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261906510 (211h39m18.99317414s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262041383 (211h39m18.993309003s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262119430 (211h39m18.993387071s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262267067 (211h39m18.993534687s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262472443 (211h39m18.993740064s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.335674416 (211h39m19.066942037s) [upload=220201]: Next returned nil segment Aug 17 13:05:38.335685137 (211h39m19.066952757s) [upload=220201]: Waiting for error group managing segments... Aug 17 13:05:38.343418127 (211h39m19.074685747s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.343638923 (211h39m19.074906542s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.375313802 (211h39m19.106581432s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.384478504 (211h39m19.115746134s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.394391995 (211h39m19.125659625s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.395010046 (211h39m19.126277676s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.402387698 (211h39m19.133655328s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.409234503 (211h39m19.140502123s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.428228938 (211h39m19.159496568s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.546929112 (211h39m19.278196732s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.562922516 (211h39m19.294190176s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.786308181 (211h39m19.517575811s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.858771665 (211h39m19.590039295s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:06:38.221240067 (211h40m18.952507717s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221311331 (211h40m18.952578981s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:104.36.82.58:4285): protocol: write tcp 5.161.89.252:35328->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221354321 (211h40m18.952621971s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: rpc: dial tcp 104.36.82.2:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221450012 (211h40m18.952717662s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:104.36.82.60:4343): protocol: rpc: dial tcp 104.36.82.60:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221534150 (211h40m18.952801799s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: write tcp 5.161.89.252:49358->104.36.82.18:4376: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221585135 (211h40m18.952852785s) [seg_pos={0 0}, upload=220201]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 13:06:38.236359888 (211h40m18.967627539s) [upload=220201]: Done uploading segments... Aug 17 13:06:38.236387120 (211h40m18.967654769s) [upload=220201]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 411.064573ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 145.642056ms FAIL (2025-08-17 13:39:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:00:24.431681640 (211h34m5.162949261s) [upload=220113]: Starting upload Aug 17 13:00:24.431716026 (211h34m5.162983646s) [upload=220113]: Done starting upload Aug 17 13:00:24.431759107 (211h34m5.163026727s) [upload=220113]: Uploading segments... Aug 17 13:00:24.431790736 (211h34m5.163058366s) [upload=220113]: Got next segment. Inline: true Aug 17 13:00:24.431795495 (211h34m5.163063115s) [upload=220113]: Waiting for error group managing segments... Aug 17 13:00:24.431947481 (211h34m5.163215111s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:00:24.431953533 (211h34m5.163221163s) [upload=220113]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:00:24.431956157 (211h34m5.163223787s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:00:24.434700137 (211h34m5.165967787s) [upload=220113]: Done uploading segments... Aug 17 13:00:24.434709414 (211h34m5.165977034s) [upload=220113]: Upload finished. err: metaclient: Too Many Requests Duration:* 145.664299ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...XXXX..................................................] duration: 397.755607ms FAIL (2025-08-17 13:39:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 13:05:03.230244168 (211h38m43.961511788s) [upload=220194]: Starting upload Aug 17 13:05:03.230282931 (211h38m43.961550561s) [upload=220194]: Done starting upload Aug 17 13:05:03.230337775 (211h38m43.961605395s) [upload=220194]: Uploading segments... Aug 17 13:05:03.230498627 (211h38m43.961766257s) [upload=220194]: Got next segment. Inline: false Aug 17 13:05:03.230523233 (211h38m43.961790852s) [upload=220194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:03.230526769 (211h38m43.96179439s) [upload=220194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:03.253244145 (211h38m43.984511775s) [seg_pos={0 0}, upload=220194]: Begin upload segment... Aug 17 13:05:03.253284361 (211h38m43.984551981s) [seg_pos={0 0}, upload=220194]: Done begin upload segment. Aug 17 13:05:03.253314057 (211h38m43.984581687s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253326811 (211h38m43.984594431s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253379720 (211h38m43.98464735s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253678482 (211h38m43.984946102s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253743043 (211h38m43.985010673s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253961614 (211h38m43.985229234s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254079065 (211h38m43.985346696s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254257481 (211h38m43.985525111s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254408585 (211h38m43.985676204s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254523081 (211h38m43.985790721s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254695415 (211h38m43.985963045s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254849924 (211h38m43.986117554s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254965993 (211h38m43.986233613s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255073975 (211h38m43.986341606s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255232403 (211h38m43.986500033s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255467977 (211h38m43.986735607s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255719861 (211h38m43.986987481s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255961866 (211h38m43.987229496s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.323589897 (211h38m44.054857527s) [upload=220194]: Next returned nil segment Aug 17 13:05:03.323600959 (211h38m44.054868579s) [upload=220194]: Waiting for error group managing segments... Aug 17 13:05:03.351486953 (211h38m44.082754583s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.356136245 (211h38m44.087403865s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.359798501 (211h38m44.091066121s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.368712994 (211h38m44.099980613s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.451425619 (211h38m44.18269327s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.488338257 (211h38m44.219605887s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.533602960 (211h38m44.26487062s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.706703573 (211h38m44.437971194s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.845738394 (211h38m44.577006013s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.854673865 (211h38m44.585941515s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:06:03.213670588 (211h39m43.944938208s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213738616 (211h39m43.945006266s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 5.161.89.252:44086->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213807617 (211h39m43.945075236s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213844345 (211h39m43.945111965s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213873840 (211h39m43.945141471s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:16954->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213895070 (211h39m43.945162691s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213912293 (211h39m43.945179912s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213916270 (211h39m43.94518389s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213938842 (211h39m43.945206472s) [seg_pos={0 0}, upload=220194]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 13:06:03.239464369 (211h39m43.970731999s) [upload=220194]: Done uploading segments... Aug 17 13:06:03.239484097 (211h39m43.970751716s) [upload=220194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 397.777397ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........XXXXXX.............................................] duration: 411.257596ms FAIL (2025-08-17 13:38:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 411.298724ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X...XXXX.................................................] duration: 898.823919ms FAIL (2025-08-17 13:38:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 13:05:03.230244168 (211h38m43.961511788s) [upload=220194]: Starting upload Aug 17 13:05:03.230282931 (211h38m43.961550561s) [upload=220194]: Done starting upload Aug 17 13:05:03.230337775 (211h38m43.961605395s) [upload=220194]: Uploading segments... Aug 17 13:05:03.230498627 (211h38m43.961766257s) [upload=220194]: Got next segment. Inline: false Aug 17 13:05:03.230523233 (211h38m43.961790852s) [upload=220194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:03.230526769 (211h38m43.96179439s) [upload=220194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:03.253244145 (211h38m43.984511775s) [seg_pos={0 0}, upload=220194]: Begin upload segment... Aug 17 13:05:03.253284361 (211h38m43.984551981s) [seg_pos={0 0}, upload=220194]: Done begin upload segment. Aug 17 13:05:03.253314057 (211h38m43.984581687s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253326811 (211h38m43.984594431s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253379720 (211h38m43.98464735s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253678482 (211h38m43.984946102s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253743043 (211h38m43.985010673s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253961614 (211h38m43.985229234s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254079065 (211h38m43.985346696s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254257481 (211h38m43.985525111s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254408585 (211h38m43.985676204s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254523081 (211h38m43.985790721s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254695415 (211h38m43.985963045s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254849924 (211h38m43.986117554s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254965993 (211h38m43.986233613s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255073975 (211h38m43.986341606s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255232403 (211h38m43.986500033s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255467977 (211h38m43.986735607s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255719861 (211h38m43.986987481s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255961866 (211h38m43.987229496s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.323589897 (211h38m44.054857527s) [upload=220194]: Next returned nil segment Aug 17 13:05:03.323600959 (211h38m44.054868579s) [upload=220194]: Waiting for error group managing segments... Aug 17 13:05:03.351486953 (211h38m44.082754583s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.356136245 (211h38m44.087403865s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.359798501 (211h38m44.091066121s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.368712994 (211h38m44.099980613s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.451425619 (211h38m44.18269327s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.488338257 (211h38m44.219605887s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.533602960 (211h38m44.26487062s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.706703573 (211h38m44.437971194s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.845738394 (211h38m44.577006013s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.854673865 (211h38m44.585941515s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:06:03.213670588 (211h39m43.944938208s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213738616 (211h39m43.945006266s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 5.161.89.252:44086->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213807617 (211h39m43.945075236s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213844345 (211h39m43.945111965s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213873840 (211h39m43.945141471s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:16954->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213895070 (211h39m43.945162691s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213912293 (211h39m43.945179912s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213916270 (211h39m43.94518389s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213938842 (211h39m43.945206472s) [seg_pos={0 0}, upload=220194]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 13:06:03.239464369 (211h39m43.970731999s) [upload=220194]: Done uploading segments... Aug 17 13:06:03.239484097 (211h39m43.970751716s) [upload=220194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 898.85084ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 175.135976ms FAIL (2025-08-17 13:38:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:00:24.431681640 (211h34m5.162949261s) [upload=220113]: Starting upload Aug 17 13:00:24.431716026 (211h34m5.162983646s) [upload=220113]: Done starting upload Aug 17 13:00:24.431759107 (211h34m5.163026727s) [upload=220113]: Uploading segments... Aug 17 13:00:24.431790736 (211h34m5.163058366s) [upload=220113]: Got next segment. Inline: true Aug 17 13:00:24.431795495 (211h34m5.163063115s) [upload=220113]: Waiting for error group managing segments... Aug 17 13:00:24.431947481 (211h34m5.163215111s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:00:24.431953533 (211h34m5.163221163s) [upload=220113]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:00:24.431956157 (211h34m5.163223787s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:00:24.434700137 (211h34m5.165967787s) [upload=220113]: Done uploading segments... Aug 17 13:00:24.434709414 (211h34m5.165977034s) [upload=220113]: Upload finished. err: metaclient: Too Many Requests Duration:* 175.162324ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XX.XX.........................................] duration: 604.912635ms FAIL (2025-08-17 13:38:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11h39m18.963301239s) [upload=220201]: Got next segment. Inline: false Aug 17 13:05:38.232064688 (211h39m18.963332307s) [upload=220201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:38.232068765 (211h39m18.963336385s) [upload=220201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:38.260394616 (211h39m18.991662237s) [seg_pos={0 0}, upload=220201]: Begin upload segment... Aug 17 13:05:38.260422680 (211h39m18.99169031s) [seg_pos={0 0}, upload=220201]: Done begin upload segment. Aug 17 13:05:38.260441515 (211h39m18.991709135s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260457946 (211h39m18.991725576s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260717223 (211h39m18.991984853s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260731841 (211h39m18.991999471s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260959870 (211h39m18.9922275s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260968356 (211h39m18.992235986s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261198057 (211h39m18.992465677s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261233694 (211h39m18.992501334s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261415967 (211h39m18.992683597s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261464098 (211h39m18.992731728s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261631563 (211h39m18.992899193s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261670385 (211h39m18.992938005s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261838482 (211h39m18.993106102s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261906510 (211h39m18.99317414s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262041383 (211h39m18.993309003s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262119430 (211h39m18.993387071s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262267067 (211h39m18.993534687s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262472443 (211h39m18.993740064s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.335674416 (211h39m19.066942037s) [upload=220201]: Next returned nil segment Aug 17 13:05:38.335685137 (211h39m19.066952757s) [upload=220201]: Waiting for error group managing segments... Aug 17 13:05:38.343418127 (211h39m19.074685747s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.343638923 (211h39m19.074906542s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.375313802 (211h39m19.106581432s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.384478504 (211h39m19.115746134s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.394391995 (211h39m19.125659625s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.395010046 (211h39m19.126277676s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.402387698 (211h39m19.133655328s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.409234503 (211h39m19.140502123s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.428228938 (211h39m19.159496568s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.546929112 (211h39m19.278196732s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.562922516 (211h39m19.294190176s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.786308181 (211h39m19.517575811s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.858771665 (211h39m19.590039295s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:06:38.221240067 (211h40m18.952507717s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221311331 (211h40m18.952578981s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:104.36.82.58:4285): protocol: write tcp 5.161.89.252:35328->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221354321 (211h40m18.952621971s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: rpc: dial tcp 104.36.82.2:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221450012 (211h40m18.952717662s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:104.36.82.60:4343): protocol: rpc: dial tcp 104.36.82.60:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221534150 (211h40m18.952801799s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: write tcp 5.161.89.252:49358->104.36.82.18:4376: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221585135 (211h40m18.952852785s) [seg_pos={0 0}, upload=220201]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 13:06:38.236359888 (211h40m18.967627539s) [upload=220201]: Done uploading segments... Aug 17 13:06:38.236387120 (211h40m18.967654769s) [upload=220201]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 604.978078ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X...XXXX................................................] duration: 414.842181ms FAIL (2025-08-17 13:38:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 13:05:03.230244168 (211h38m43.961511788s) [upload=220194]: Starting upload Aug 17 13:05:03.230282931 (211h38m43.961550561s) [upload=220194]: Done starting upload Aug 17 13:05:03.230337775 (211h38m43.961605395s) [upload=220194]: Uploading segments... Aug 17 13:05:03.230498627 (211h38m43.961766257s) [upload=220194]: Got next segment. Inline: false Aug 17 13:05:03.230523233 (211h38m43.961790852s) [upload=220194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:03.230526769 (211h38m43.96179439s) [upload=220194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:03.253244145 (211h38m43.984511775s) [seg_pos={0 0}, upload=220194]: Begin upload segment... Aug 17 13:05:03.253284361 (211h38m43.984551981s) [seg_pos={0 0}, upload=220194]: Done begin upload segment. Aug 17 13:05:03.253314057 (211h38m43.984581687s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253326811 (211h38m43.984594431s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253379720 (211h38m43.98464735s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253678482 (211h38m43.984946102s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253743043 (211h38m43.985010673s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253961614 (211h38m43.985229234s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254079065 (211h38m43.985346696s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254257481 (211h38m43.985525111s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254408585 (211h38m43.985676204s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254523081 (211h38m43.985790721s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254695415 (211h38m43.985963045s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254849924 (211h38m43.986117554s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254965993 (211h38m43.986233613s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255073975 (211h38m43.986341606s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255232403 (211h38m43.986500033s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255467977 (211h38m43.986735607s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255719861 (211h38m43.986987481s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255961866 (211h38m43.987229496s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.323589897 (211h38m44.054857527s) [upload=220194]: Next returned nil segment Aug 17 13:05:03.323600959 (211h38m44.054868579s) [upload=220194]: Waiting for error group managing segments... Aug 17 13:05:03.351486953 (211h38m44.082754583s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.356136245 (211h38m44.087403865s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.359798501 (211h38m44.091066121s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.368712994 (211h38m44.099980613s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.451425619 (211h38m44.18269327s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.488338257 (211h38m44.219605887s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.533602960 (211h38m44.26487062s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.706703573 (211h38m44.437971194s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.845738394 (211h38m44.577006013s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.854673865 (211h38m44.585941515s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:06:03.213670588 (211h39m43.944938208s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213738616 (211h39m43.945006266s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 5.161.89.252:44086->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213807617 (211h39m43.945075236s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213844345 (211h39m43.945111965s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213873840 (211h39m43.945141471s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:16954->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213895070 (211h39m43.945162691s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213912293 (211h39m43.945179912s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213916270 (211h39m43.94518389s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213938842 (211h39m43.945206472s) [seg_pos={0 0}, upload=220194]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 13:06:03.239464369 (211h39m43.970731999s) [upload=220194]: Done uploading segments... Aug 17 13:06:03.239484097 (211h39m43.970751716s) [upload=220194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 414.863792ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........XXXXXX............................................] duration: 371.85739ms FAIL (2025-08-17 13:38:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 371.878039ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 194.776321ms FAIL (2025-08-17 13:38:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:00:24.431681640 (211h34m5.162949261s) [upload=220113]: Starting upload Aug 17 13:00:24.431716026 (211h34m5.162983646s) [upload=220113]: Done starting upload Aug 17 13:00:24.431759107 (211h34m5.163026727s) [upload=220113]: Uploading segments... Aug 17 13:00:24.431790736 (211h34m5.163058366s) [upload=220113]: Got next segment. Inline: true Aug 17 13:00:24.431795495 (211h34m5.163063115s) [upload=220113]: Waiting for error group managing segments... Aug 17 13:00:24.431947481 (211h34m5.163215111s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:00:24.431953533 (211h34m5.163221163s) [upload=220113]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:00:24.431956157 (211h34m5.163223787s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:00:24.434700137 (211h34m5.165967787s) [upload=220113]: Done uploading segments... Aug 17 13:00:24.434709414 (211h34m5.165977034s) [upload=220113]: Upload finished. err: metaclient: Too Many Requests Duration:* 194.798163ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XX.XX........................................] duration: 464.349ms FAIL (2025-08-17 13:38:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11h39m18.963301239s) [upload=220201]: Got next segment. Inline: false Aug 17 13:05:38.232064688 (211h39m18.963332307s) [upload=220201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:38.232068765 (211h39m18.963336385s) [upload=220201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:38.260394616 (211h39m18.991662237s) [seg_pos={0 0}, upload=220201]: Begin upload segment... Aug 17 13:05:38.260422680 (211h39m18.99169031s) [seg_pos={0 0}, upload=220201]: Done begin upload segment. Aug 17 13:05:38.260441515 (211h39m18.991709135s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260457946 (211h39m18.991725576s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260717223 (211h39m18.991984853s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260731841 (211h39m18.991999471s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260959870 (211h39m18.9922275s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260968356 (211h39m18.992235986s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261198057 (211h39m18.992465677s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261233694 (211h39m18.992501334s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261415967 (211h39m18.992683597s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261464098 (211h39m18.992731728s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261631563 (211h39m18.992899193s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261670385 (211h39m18.992938005s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261838482 (211h39m18.993106102s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261906510 (211h39m18.99317414s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262041383 (211h39m18.993309003s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262119430 (211h39m18.993387071s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262267067 (211h39m18.993534687s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262472443 (211h39m18.993740064s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.335674416 (211h39m19.066942037s) [upload=220201]: Next returned nil segment Aug 17 13:05:38.335685137 (211h39m19.066952757s) [upload=220201]: Waiting for error group managing segments... Aug 17 13:05:38.343418127 (211h39m19.074685747s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.343638923 (211h39m19.074906542s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.375313802 (211h39m19.106581432s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.384478504 (211h39m19.115746134s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.394391995 (211h39m19.125659625s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.395010046 (211h39m19.126277676s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.402387698 (211h39m19.133655328s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.409234503 (211h39m19.140502123s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.428228938 (211h39m19.159496568s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.546929112 (211h39m19.278196732s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.562922516 (211h39m19.294190176s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.786308181 (211h39m19.517575811s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.858771665 (211h39m19.590039295s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:06:38.221240067 (211h40m18.952507717s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221311331 (211h40m18.952578981s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:104.36.82.58:4285): protocol: write tcp 5.161.89.252:35328->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221354321 (211h40m18.952621971s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: rpc: dial tcp 104.36.82.2:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221450012 (211h40m18.952717662s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:104.36.82.60:4343): protocol: rpc: dial tcp 104.36.82.60:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221534150 (211h40m18.952801799s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: write tcp 5.161.89.252:49358->104.36.82.18:4376: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221585135 (211h40m18.952852785s) [seg_pos={0 0}, upload=220201]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 13:06:38.236359888 (211h40m18.967627539s) [upload=220201]: Done uploading segments... Aug 17 13:06:38.236387120 (211h40m18.967654769s) [upload=220201]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 464.383306ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X...XXXX...............................................] duration: 559.351345ms FAIL (2025-08-17 13:38:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 13:05:03.230244168 (211h38m43.961511788s) [upload=220194]: Starting upload Aug 17 13:05:03.230282931 (211h38m43.961550561s) [upload=220194]: Done starting upload Aug 17 13:05:03.230337775 (211h38m43.961605395s) [upload=220194]: Uploading segments... Aug 17 13:05:03.230498627 (211h38m43.961766257s) [upload=220194]: Got next segment. Inline: false Aug 17 13:05:03.230523233 (211h38m43.961790852s) [upload=220194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:03.230526769 (211h38m43.96179439s) [upload=220194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:03.253244145 (211h38m43.984511775s) [seg_pos={0 0}, upload=220194]: Begin upload segment... Aug 17 13:05:03.253284361 (211h38m43.984551981s) [seg_pos={0 0}, upload=220194]: Done begin upload segment. Aug 17 13:05:03.253314057 (211h38m43.984581687s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253326811 (211h38m43.984594431s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253379720 (211h38m43.98464735s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253678482 (211h38m43.984946102s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253743043 (211h38m43.985010673s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253961614 (211h38m43.985229234s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254079065 (211h38m43.985346696s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254257481 (211h38m43.985525111s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254408585 (211h38m43.985676204s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254523081 (211h38m43.985790721s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254695415 (211h38m43.985963045s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254849924 (211h38m43.986117554s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254965993 (211h38m43.986233613s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255073975 (211h38m43.986341606s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255232403 (211h38m43.986500033s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255467977 (211h38m43.986735607s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255719861 (211h38m43.986987481s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255961866 (211h38m43.987229496s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.323589897 (211h38m44.054857527s) [upload=220194]: Next returned nil segment Aug 17 13:05:03.323600959 (211h38m44.054868579s) [upload=220194]: Waiting for error group managing segments... Aug 17 13:05:03.351486953 (211h38m44.082754583s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.356136245 (211h38m44.087403865s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.359798501 (211h38m44.091066121s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.368712994 (211h38m44.099980613s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.451425619 (211h38m44.18269327s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.488338257 (211h38m44.219605887s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.533602960 (211h38m44.26487062s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.706703573 (211h38m44.437971194s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.845738394 (211h38m44.577006013s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.854673865 (211h38m44.585941515s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:06:03.213670588 (211h39m43.944938208s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213738616 (211h39m43.945006266s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 5.161.89.252:44086->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213807617 (211h39m43.945075236s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213844345 (211h39m43.945111965s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213873840 (211h39m43.945141471s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:16954->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213895070 (211h39m43.945162691s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213912293 (211h39m43.945179912s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213916270 (211h39m43.94518389s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213938842 (211h39m43.945206472s) [seg_pos={0 0}, upload=220194]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 13:06:03.239464369 (211h39m43.970731999s) [upload=220194]: Done uploading segments... Aug 17 13:06:03.239484097 (211h39m43.970751716s) [upload=220194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 559.373768ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........XXXXXX...........................................] duration: 381.411021ms FAIL (2025-08-17 13:37:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 381.439536ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 232.675484ms FAIL (2025-08-17 13:37:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:00:24.431681640 (211h34m5.162949261s) [upload=220113]: Starting upload Aug 17 13:00:24.431716026 (211h34m5.162983646s) [upload=220113]: Done starting upload Aug 17 13:00:24.431759107 (211h34m5.163026727s) [upload=220113]: Uploading segments... Aug 17 13:00:24.431790736 (211h34m5.163058366s) [upload=220113]: Got next segment. Inline: true Aug 17 13:00:24.431795495 (211h34m5.163063115s) [upload=220113]: Waiting for error group managing segments... Aug 17 13:00:24.431947481 (211h34m5.163215111s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:00:24.431953533 (211h34m5.163221163s) [upload=220113]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:00:24.431956157 (211h34m5.163223787s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:00:24.434700137 (211h34m5.165967787s) [upload=220113]: Done uploading segments... Aug 17 13:00:24.434709414 (211h34m5.165977034s) [upload=220113]: Upload finished. err: metaclient: Too Many Requests Duration:* 232.702404ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............XXXXXX..........................................] duration: 634.711084ms FAIL (2025-08-17 13:37:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 634.734087ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XX.XX.......................................] duration: 442.690563ms FAIL (2025-08-17 13:36:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11h39m18.963301239s) [upload=220201]: Got next segment. Inline: false Aug 17 13:05:38.232064688 (211h39m18.963332307s) [upload=220201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:38.232068765 (211h39m18.963336385s) [upload=220201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:38.260394616 (211h39m18.991662237s) [seg_pos={0 0}, upload=220201]: Begin upload segment... Aug 17 13:05:38.260422680 (211h39m18.99169031s) [seg_pos={0 0}, upload=220201]: Done begin upload segment. Aug 17 13:05:38.260441515 (211h39m18.991709135s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260457946 (211h39m18.991725576s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260717223 (211h39m18.991984853s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260731841 (211h39m18.991999471s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260959870 (211h39m18.9922275s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260968356 (211h39m18.992235986s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261198057 (211h39m18.992465677s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261233694 (211h39m18.992501334s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261415967 (211h39m18.992683597s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261464098 (211h39m18.992731728s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261631563 (211h39m18.992899193s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261670385 (211h39m18.992938005s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261838482 (211h39m18.993106102s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261906510 (211h39m18.99317414s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262041383 (211h39m18.993309003s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262119430 (211h39m18.993387071s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262267067 (211h39m18.993534687s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262472443 (211h39m18.993740064s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.335674416 (211h39m19.066942037s) [upload=220201]: Next returned nil segment Aug 17 13:05:38.335685137 (211h39m19.066952757s) [upload=220201]: Waiting for error group managing segments... Aug 17 13:05:38.343418127 (211h39m19.074685747s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.343638923 (211h39m19.074906542s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.375313802 (211h39m19.106581432s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.384478504 (211h39m19.115746134s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.394391995 (211h39m19.125659625s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.395010046 (211h39m19.126277676s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.402387698 (211h39m19.133655328s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.409234503 (211h39m19.140502123s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.428228938 (211h39m19.159496568s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.546929112 (211h39m19.278196732s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.562922516 (211h39m19.294190176s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.786308181 (211h39m19.517575811s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.858771665 (211h39m19.590039295s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:06:38.221240067 (211h40m18.952507717s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221311331 (211h40m18.952578981s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:104.36.82.58:4285): protocol: write tcp 5.161.89.252:35328->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221354321 (211h40m18.952621971s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: rpc: dial tcp 104.36.82.2:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221450012 (211h40m18.952717662s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:104.36.82.60:4343): protocol: rpc: dial tcp 104.36.82.60:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221534150 (211h40m18.952801799s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: write tcp 5.161.89.252:49358->104.36.82.18:4376: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221585135 (211h40m18.952852785s) [seg_pos={0 0}, upload=220201]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 13:06:38.236359888 (211h40m18.967627539s) [upload=220201]: Done uploading segments... Aug 17 13:06:38.236387120 (211h40m18.967654769s) [upload=220201]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 442.736208ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X...XXXX..............................................] duration: 671.407636ms FAIL (2025-08-17 13:36:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 13:05:03.230244168 (211h38m43.961511788s) [upload=220194]: Starting upload Aug 17 13:05:03.230282931 (211h38m43.961550561s) [upload=220194]: Done starting upload Aug 17 13:05:03.230337775 (211h38m43.961605395s) [upload=220194]: Uploading segments... Aug 17 13:05:03.230498627 (211h38m43.961766257s) [upload=220194]: Got next segment. Inline: false Aug 17 13:05:03.230523233 (211h38m43.961790852s) [upload=220194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:03.230526769 (211h38m43.96179439s) [upload=220194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:03.253244145 (211h38m43.984511775s) [seg_pos={0 0}, upload=220194]: Begin upload segment... Aug 17 13:05:03.253284361 (211h38m43.984551981s) [seg_pos={0 0}, upload=220194]: Done begin upload segment. Aug 17 13:05:03.253314057 (211h38m43.984581687s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253326811 (211h38m43.984594431s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253379720 (211h38m43.98464735s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253678482 (211h38m43.984946102s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253743043 (211h38m43.985010673s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253961614 (211h38m43.985229234s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254079065 (211h38m43.985346696s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254257481 (211h38m43.985525111s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254408585 (211h38m43.985676204s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254523081 (211h38m43.985790721s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254695415 (211h38m43.985963045s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254849924 (211h38m43.986117554s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254965993 (211h38m43.986233613s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255073975 (211h38m43.986341606s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255232403 (211h38m43.986500033s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255467977 (211h38m43.986735607s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255719861 (211h38m43.986987481s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255961866 (211h38m43.987229496s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.323589897 (211h38m44.054857527s) [upload=220194]: Next returned nil segment Aug 17 13:05:03.323600959 (211h38m44.054868579s) [upload=220194]: Waiting for error group managing segments... Aug 17 13:05:03.351486953 (211h38m44.082754583s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.356136245 (211h38m44.087403865s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.359798501 (211h38m44.091066121s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.368712994 (211h38m44.099980613s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.451425619 (211h38m44.18269327s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.488338257 (211h38m44.219605887s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.533602960 (211h38m44.26487062s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.706703573 (211h38m44.437971194s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.845738394 (211h38m44.577006013s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.854673865 (211h38m44.585941515s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:06:03.213670588 (211h39m43.944938208s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213738616 (211h39m43.945006266s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 5.161.89.252:44086->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213807617 (211h39m43.945075236s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213844345 (211h39m43.945111965s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213873840 (211h39m43.945141471s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:16954->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213895070 (211h39m43.945162691s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213912293 (211h39m43.945179912s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213916270 (211h39m43.94518389s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213938842 (211h39m43.945206472s) [seg_pos={0 0}, upload=220194]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 13:06:03.239464369 (211h39m43.970731999s) [upload=220194]: Done uploading segments... Aug 17 13:06:03.239484097 (211h39m43.970751716s) [upload=220194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 671.423385ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 574.419269ms FAIL (2025-08-17 13:36:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:00:24.431681640 (211h34m5.162949261s) [upload=220113]: Starting upload Aug 17 13:00:24.431716026 (211h34m5.162983646s) [upload=220113]: Done starting upload Aug 17 13:00:24.431759107 (211h34m5.163026727s) [upload=220113]: Uploading segments... Aug 17 13:00:24.431790736 (211h34m5.163058366s) [upload=220113]: Got next segment. Inline: true Aug 17 13:00:24.431795495 (211h34m5.163063115s) [upload=220113]: Waiting for error group managing segments... Aug 17 13:00:24.431947481 (211h34m5.163215111s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:00:24.431953533 (211h34m5.163221163s) [upload=220113]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:00:24.431956157 (211h34m5.163223787s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:00:24.434700137 (211h34m5.165967787s) [upload=220113]: Done uploading segments... Aug 17 13:00:24.434709414 (211h34m5.165977034s) [upload=220113]: Upload finished. err: metaclient: Too Many Requests Duration:* 574.444808ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............XXXXXX.........................................] duration: 354.782715ms FAIL (2025-08-17 13:36:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 354.815156ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XX.XX......................................] duration: 1.029460311s FAIL (2025-08-17 13:36:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11h39m18.963301239s) [upload=220201]: Got next segment. Inline: false Aug 17 13:05:38.232064688 (211h39m18.963332307s) [upload=220201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:38.232068765 (211h39m18.963336385s) [upload=220201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:38.260394616 (211h39m18.991662237s) [seg_pos={0 0}, upload=220201]: Begin upload segment... Aug 17 13:05:38.260422680 (211h39m18.99169031s) [seg_pos={0 0}, upload=220201]: Done begin upload segment. Aug 17 13:05:38.260441515 (211h39m18.991709135s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260457946 (211h39m18.991725576s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260717223 (211h39m18.991984853s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260731841 (211h39m18.991999471s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260959870 (211h39m18.9922275s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260968356 (211h39m18.992235986s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261198057 (211h39m18.992465677s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261233694 (211h39m18.992501334s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261415967 (211h39m18.992683597s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261464098 (211h39m18.992731728s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261631563 (211h39m18.992899193s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261670385 (211h39m18.992938005s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261838482 (211h39m18.993106102s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261906510 (211h39m18.99317414s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262041383 (211h39m18.993309003s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262119430 (211h39m18.993387071s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262267067 (211h39m18.993534687s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262472443 (211h39m18.993740064s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.335674416 (211h39m19.066942037s) [upload=220201]: Next returned nil segment Aug 17 13:05:38.335685137 (211h39m19.066952757s) [upload=220201]: Waiting for error group managing segments... Aug 17 13:05:38.343418127 (211h39m19.074685747s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.343638923 (211h39m19.074906542s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.375313802 (211h39m19.106581432s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.384478504 (211h39m19.115746134s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.394391995 (211h39m19.125659625s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.395010046 (211h39m19.126277676s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.402387698 (211h39m19.133655328s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.409234503 (211h39m19.140502123s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.428228938 (211h39m19.159496568s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.546929112 (211h39m19.278196732s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.562922516 (211h39m19.294190176s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.786308181 (211h39m19.517575811s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.858771665 (211h39m19.590039295s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:06:38.221240067 (211h40m18.952507717s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221311331 (211h40m18.952578981s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:104.36.82.58:4285): protocol: write tcp 5.161.89.252:35328->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221354321 (211h40m18.952621971s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: rpc: dial tcp 104.36.82.2:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221450012 (211h40m18.952717662s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:104.36.82.60:4343): protocol: rpc: dial tcp 104.36.82.60:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221534150 (211h40m18.952801799s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: write tcp 5.161.89.252:49358->104.36.82.18:4376: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221585135 (211h40m18.952852785s) [seg_pos={0 0}, upload=220201]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 13:06:38.236359888 (211h40m18.967627539s) [upload=220201]: Done uploading segments... Aug 17 13:06:38.236387120 (211h40m18.967654769s) [upload=220201]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.029484767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...XXXX.............................................] duration: 599.625564ms FAIL (2025-08-17 13:36:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 13:05:03.230244168 (211h38m43.961511788s) [upload=220194]: Starting upload Aug 17 13:05:03.230282931 (211h38m43.961550561s) [upload=220194]: Done starting upload Aug 17 13:05:03.230337775 (211h38m43.961605395s) [upload=220194]: Uploading segments... Aug 17 13:05:03.230498627 (211h38m43.961766257s) [upload=220194]: Got next segment. Inline: false Aug 17 13:05:03.230523233 (211h38m43.961790852s) [upload=220194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:03.230526769 (211h38m43.96179439s) [upload=220194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:03.253244145 (211h38m43.984511775s) [seg_pos={0 0}, upload=220194]: Begin upload segment... Aug 17 13:05:03.253284361 (211h38m43.984551981s) [seg_pos={0 0}, upload=220194]: Done begin upload segment. Aug 17 13:05:03.253314057 (211h38m43.984581687s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253326811 (211h38m43.984594431s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253379720 (211h38m43.98464735s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253678482 (211h38m43.984946102s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253743043 (211h38m43.985010673s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253961614 (211h38m43.985229234s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254079065 (211h38m43.985346696s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254257481 (211h38m43.985525111s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254408585 (211h38m43.985676204s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254523081 (211h38m43.985790721s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254695415 (211h38m43.985963045s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254849924 (211h38m43.986117554s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254965993 (211h38m43.986233613s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255073975 (211h38m43.986341606s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255232403 (211h38m43.986500033s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255467977 (211h38m43.986735607s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255719861 (211h38m43.986987481s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255961866 (211h38m43.987229496s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.323589897 (211h38m44.054857527s) [upload=220194]: Next returned nil segment Aug 17 13:05:03.323600959 (211h38m44.054868579s) [upload=220194]: Waiting for error group managing segments... Aug 17 13:05:03.351486953 (211h38m44.082754583s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.356136245 (211h38m44.087403865s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.359798501 (211h38m44.091066121s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.368712994 (211h38m44.099980613s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.451425619 (211h38m44.18269327s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.488338257 (211h38m44.219605887s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.533602960 (211h38m44.26487062s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.706703573 (211h38m44.437971194s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.845738394 (211h38m44.577006013s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.854673865 (211h38m44.585941515s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:06:03.213670588 (211h39m43.944938208s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213738616 (211h39m43.945006266s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 5.161.89.252:44086->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213807617 (211h39m43.945075236s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213844345 (211h39m43.945111965s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213873840 (211h39m43.945141471s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:16954->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213895070 (211h39m43.945162691s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213912293 (211h39m43.945179912s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213916270 (211h39m43.94518389s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213938842 (211h39m43.945206472s) [seg_pos={0 0}, upload=220194]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 13:06:03.239464369 (211h39m43.970731999s) [upload=220194]: Done uploading segments... Aug 17 13:06:03.239484097 (211h39m43.970751716s) [upload=220194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 599.653014ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............XXXXXX........................................] duration: 393.977135ms FAIL (2025-08-17 13:36:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 393.990872ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............XXXXXX.......................................] duration: 556.170518ms FAIL (2025-08-17 13:35:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Aug 17 13:05:26.773910806 (211h39m7.505178455s) [upload=220199]: Starting upload Aug 17 13:05:26.774013439 (211h39m7.505281089s) [upload=220199]: Done starting upload Aug 17 13:05:26.774108687 (211h39m7.505376347s) [upload=220199]: Uploading segments... Aug 17 13:05:26.774448908 (211h39m7.505716557s) [upload=220199]: Got next segment. Inline: false Aug 17 13:05:26.774529659 (211h39m7.505797309s) [upload=220199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:26.774542563 (211h39m7.505810413s) [upload=220199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:26.800669942 (211h39m7.531937602s) [seg_pos={0 0}, upload=220199]: Begin upload segment... Aug 17 13:05:26.800736297 (211h39m7.532003946s) [seg_pos={0 0}, upload=220199]: Done begin upload segment. Aug 17 13:05:26.800841635 (211h39m7.532109516s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800868956 (211h39m7.532136666s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.800950009 (211h39m7.532217658s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801393953 (211h39m7.532661603s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801475126 (211h39m7.532742776s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.801832077 (211h39m7.533099726s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802104890 (211h39m7.53337254s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802321588 (211h39m7.533589238s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802678157 (211h39m7.533945808s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.802690742 (211h39m7.533958392s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803042964 (211h39m7.534310614s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803191042 (211h39m7.534458693s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.803742099 (211h39m7.535009759s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804353918 (211h39m7.535621578s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.804900065 (211h39m7.536167716s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.805497930 (211h39m7.53676559s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Uploading piece... Aug 17 13:05:26.903223044 (211h39m7.634490693s) [upload=220199]: Next returned nil segment Aug 17 13:05:26.903279810 (211h39m7.63454746s) [upload=220199]: Waiting for error group managing segments... Aug 17 13:05:26.911780204 (211h39m7.643047853s) [noise=true , address=45.91.116.9:4262 , piece_id=ZTTEWA5X5O5WO5H7IFXOTYDIY7CVBEQMTAPTHNDJO2UZE7QVFM3Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912017279 (211h39m7.643284919s) [noise=true , address=45.91.116.6:4974 , piece_id=HLURL4RNIP2HSHV5TETIKIPLJKNOZKBR2SLRIKZCGY2MWWI7HABA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912520456 (211h39m7.643788106s) [noise=true , address=45.91.116.4:4940 , piece_id=QLTDLLP275BJQAN2CFMVNIK4K5NYR2ZGUV36HIERRCCFGAN6VH6Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.912758152 (211h39m7.644025793s) [noise=true , address=45.91.116.14:4191 , piece_id=VEMVIKQ666V2QWJQITHQHSBUQ6QLGLBIOJ2XQF2IXBO7BMMVTBWQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.913062364 (211h39m7.644330014s) [noise=true , address=45.91.116.9:4261 , piece_id=KBGNDO3RZHBHAWY2ETPNTZ5HDSGILE5X2NFYGLOMWMYPOVXWVUDA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.927349949 (211h39m7.6586176s) [noise=true , address=77.90.31.11:4555 , piece_id=FCHMEA7RHDWWOO3NAUKXKVVNT37MP6RQGZW4BD55IPNRLMEAK4AQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.929707954 (211h39m7.660975594s) [noise=true , address=45.91.116.14:4190 , piece_id=LC3C2G2UESMRQRTZMDYQKM54QDET22XUTADEGP2YGITU5YYXPBYA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.934481991 (211h39m7.665749641s) [noise=true , address=77.90.31.16:4733 , piece_id=TINS7Y5RCDRQUWZYYTV3IBHIZ2PG6LNITWVSEC4SL54IQKSMC5QA, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:26.937408663 (211h39m7.668676314s) [noise=true , address=77.90.31.18:4805 , piece_id=BKSD3HD6OUJZKB6TJG67TNPVVGVRN2GVQDQA34PNEPJAARFUKDOQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: <nil> Aug 17 13:05:56.761492801 (211h39m37.492760422s) [noise=true , address=104.36.82.8:4970 , piece_id=2Y34EPJ4LLNB4VUSI7TX6SPRDP2F256PUAHP5TACFXMFJJD57S5A, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761561431 (211h39m37.492829051s) [noise=true , address=104.36.82.15:4257 , piece_id=PQ7DGH24VD6XRZ63IFUMTLFII23H5XZF7NBIR4FDIJXMRGLTIC5Q, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761599212 (211h39m37.492866831s) [noise=true , address=104.36.82.9:4005 , piece_id=RKXWGUGRFMSMGUEZQOMOQTYVTUJTA7WARJRRUEJJSOIXMVON7KEQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761667450 (211h39m37.49293507s) [noise=true , address=104.36.82.12:4149 , piece_id=PQWQO44XQSYBHQG74FAOIPN2LKGYQ4BJK6UX3DI3YGE7T242ADKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761701605 (211h39m37.492969224s) [noise=true , address=104.36.82.7:4910 , piece_id=2OWRVMHDDCWIAAETXMFAOSIVF5FXXTFFJ24WGDBFE6BDHGGEWIKQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761733755 (211h39m37.493001385s) [noise=true , address=104.36.82.5:4838 , piece_id=G3ZEAN7JTYRAI7LV5KDUE6W3CNLQHC4FNPODY76YKIWH52PJ7GBQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:05:56.761762949 (211h39m37.493030579s) [noise=true , address=104.36.82.2:4732 , piece_id=FTNHKCT5PLYXBMXLS4CA4ZGQYJ2A7QLH3TJI36NYVU5YXBGGEXDQ, seg_pos={0 0}, upload=220199]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 13:05:56.761788137 (211h39m37.493055757s) [seg_pos={0 0}, upload=220199]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 17 13:05:56.776656205 (211h39m37.507923835s) [upload=220199]: Done uploading segments... Aug 17 13:05:56.776672235 (211h39m37.507939865s) [upload=220199]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 556.195385ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 166.852811ms FAIL (2025-08-17 13:35:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:00:24.431681640 (211h34m5.162949261s) [upload=220113]: Starting upload Aug 17 13:00:24.431716026 (211h34m5.162983646s) [upload=220113]: Done starting upload Aug 17 13:00:24.431759107 (211h34m5.163026727s) [upload=220113]: Uploading segments... Aug 17 13:00:24.431790736 (211h34m5.163058366s) [upload=220113]: Got next segment. Inline: true Aug 17 13:00:24.431795495 (211h34m5.163063115s) [upload=220113]: Waiting for error group managing segments... Aug 17 13:00:24.431947481 (211h34m5.163215111s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:00:24.431953533 (211h34m5.163221163s) [upload=220113]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:00:24.431956157 (211h34m5.163223787s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:00:24.434700137 (211h34m5.165967787s) [upload=220113]: Done uploading segments... Aug 17 13:00:24.434709414 (211h34m5.165977034s) [upload=220113]: Upload finished. err: metaclient: Too Many Requests Duration:* 166.891844ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................XX.XX.....................................] duration: 497.452892ms FAIL (2025-08-17 13:35:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11h39m18.963301239s) [upload=220201]: Got next segment. Inline: false Aug 17 13:05:38.232064688 (211h39m18.963332307s) [upload=220201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:38.232068765 (211h39m18.963336385s) [upload=220201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:38.260394616 (211h39m18.991662237s) [seg_pos={0 0}, upload=220201]: Begin upload segment... Aug 17 13:05:38.260422680 (211h39m18.99169031s) [seg_pos={0 0}, upload=220201]: Done begin upload segment. Aug 17 13:05:38.260441515 (211h39m18.991709135s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260457946 (211h39m18.991725576s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260717223 (211h39m18.991984853s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260731841 (211h39m18.991999471s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260959870 (211h39m18.9922275s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.260968356 (211h39m18.992235986s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261198057 (211h39m18.992465677s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261233694 (211h39m18.992501334s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261415967 (211h39m18.992683597s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261464098 (211h39m18.992731728s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261631563 (211h39m18.992899193s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261670385 (211h39m18.992938005s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261838482 (211h39m18.993106102s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.261906510 (211h39m18.99317414s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262041383 (211h39m18.993309003s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262119430 (211h39m18.993387071s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262267067 (211h39m18.993534687s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.262472443 (211h39m18.993740064s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Uploading piece... Aug 17 13:05:38.335674416 (211h39m19.066942037s) [upload=220201]: Next returned nil segment Aug 17 13:05:38.335685137 (211h39m19.066952757s) [upload=220201]: Waiting for error group managing segments... Aug 17 13:05:38.343418127 (211h39m19.074685747s) [noise=true , address=79.127.219.48:4101 , piece_id=FE5LSQUF7N42F5OZMKOVGT6YJY52SORGTVK6L432JIMIHXNSC6VQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.343638923 (211h39m19.074906542s) [noise=true , address=45.91.116.14:4192 , piece_id=3WIX7ZA43BIBVVZECKYAYSTRIH3VMPCK2OPB27TRFHZYPBUTJHKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.375313802 (211h39m19.106581432s) [noise=true , address=45.91.116.13:4173 , piece_id=NGIQVVZMXTHDYHUQUK3CKN5MMEKZBXQGAKYXZ6Q3AFSW7RBHQS6A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.384478504 (211h39m19.115746134s) [noise=true , address=77.90.31.12:4597 , piece_id=KEM4R5A2NNSX454JM4Z6YJZD2U7DXNCVFMSTSPULFAAWXA5TWSAQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.394391995 (211h39m19.125659625s) [noise=true , address=77.90.31.21:4882 , piece_id=HWQGRTQH45S4PLH37OZCMPRTQAOTWC6XWDU2MQKGTSMYZE3JU2TQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.395010046 (211h39m19.126277676s) [noise=true , address=45.91.116.10:4103 , piece_id=NOIXWLDGZHHCBFH74MNE2QYY7HUWYKE6FM7RADI2WMKW3NJZ7XYQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.402387698 (211h39m19.133655328s) [noise=true , address=45.91.116.11:4132 , piece_id=WG3XB5RFLVKWWTJOHMZIOLRCZI76TP6B5RSCXMIYAJ6YJ4CEIC6Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.409234503 (211h39m19.140502123s) [noise=true , address=45.91.116.7:4042 , piece_id=46YAQQV724C3JIUJNX7MI4WUDNVJZJYMG2RKQZN7TVLWKPBOJ22Q, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.428228938 (211h39m19.159496568s) [noise=true , address=79.127.220.97:4385 , piece_id=3NKAFBP245MT6PDEJ6EWBVZNEYJZ7MYAIGWFDPFEFXF47EZ4EW7A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.546929112 (211h39m19.278196732s) [noise=true , address=79.127.201.215:4356 , piece_id=LP3WWFCR4U54H5BUGUMOHAYZCAP7GKVWCCM7LMSIQO3XZEU6BHMQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.562922516 (211h39m19.294190176s) [noise=true , address=77.90.31.13:4625 , piece_id=JNAHBL4CT76MH6Z2MBL6DR4GFWHE6KXSYCW7PGA3R2JGCA4R4A3A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.786308181 (211h39m19.517575811s) [noise=true , address=79.127.205.233:4648 , piece_id=ISW4FQMK6FB45FXYFZGSAYUHDM7KRXRC2JEXIGC2ETC64NAWO6BA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:05:38.858771665 (211h39m19.590039295s) [noise=true , address=66.23.193.175:4563 , piece_id=ULXXDJ2UQHHUPBKSJBRL3VV2M6BTDMICQZG3NENPCX5235BMNVKQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: <nil> Aug 17 13:06:38.221240067 (211h40m18.952507717s) [noise=true , address=104.36.82.11:4122 , piece_id=WSGK65J4MUUMLDG2RBNSG3MMRI2IXEHVNPY7JGO6AJGTA4I6YARA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221311331 (211h40m18.952578981s) [noise=true , address=104.36.82.58:4285 , piece_id=OQJLZYBUZ7SSYOYGPRUI6TAKAVDFZBFL7MKXIZWQPPHDSFNZFYIA, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp, address:104.36.82.58:4285): protocol: write tcp 5.161.89.252:35328->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221354321 (211h40m18.952621971s) [noise=true , address=104.36.82.2:4759 , piece_id=WKBOMTCR5U46DYEUJL65GNZVT3TXZP7QFEL6TJ6PYBGLAIRRQWUQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:104.36.82.2:4759): protocol: rpc: dial tcp 104.36.82.2:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221450012 (211h40m18.952717662s) [noise=true , address=104.36.82.60:4343 , piece_id=NUKCLM5LQEACFEXGHTGLSAJTFAXQ7PN6EYB4UUUDJC64RKNKSS4A, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:104.36.82.60:4343): protocol: rpc: dial tcp 104.36.82.60:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221534150 (211h40m18.952801799s) [noise=true , address=104.36.82.18:4376 , piece_id=6ODCXHQ7GIV3IHNCXEPEWBORVJJG644BCWTQE3LQDIQF3LT2JLDQ, seg_pos={0 0}, upload=220201]: Done uploading piece. err: ecclient: upload failed (node:12QfYPc1vbgwcnEWk67Bfbt5KiQ8sq322VKe951PFMTF6VaTzAR, address:104.36.82.18:4376): protocol: write tcp 5.161.89.252:49358->104.36.82.18:4376: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:38.221585135 (211h40m18.952852785s) [seg_pos={0 0}, upload=220201]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 13:06:38.236359888 (211h40m18.967627539s) [upload=220201]: Done uploading segments... Aug 17 13:06:38.236387120 (211h40m18.967654769s) [upload=220201]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 497.474953ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X...XXXX............................................] duration: 778.668821ms FAIL (2025-08-17 13:35:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 17 13:05:03.230244168 (211h38m43.961511788s) [upload=220194]: Starting upload Aug 17 13:05:03.230282931 (211h38m43.961550561s) [upload=220194]: Done starting upload Aug 17 13:05:03.230337775 (211h38m43.961605395s) [upload=220194]: Uploading segments... Aug 17 13:05:03.230498627 (211h38m43.961766257s) [upload=220194]: Got next segment. Inline: false Aug 17 13:05:03.230523233 (211h38m43.961790852s) [upload=220194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:05:03.230526769 (211h38m43.96179439s) [upload=220194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 13:05:03.253244145 (211h38m43.984511775s) [seg_pos={0 0}, upload=220194]: Begin upload segment... Aug 17 13:05:03.253284361 (211h38m43.984551981s) [seg_pos={0 0}, upload=220194]: Done begin upload segment. Aug 17 13:05:03.253314057 (211h38m43.984581687s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253326811 (211h38m43.984594431s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253379720 (211h38m43.98464735s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253678482 (211h38m43.984946102s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253743043 (211h38m43.985010673s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.253961614 (211h38m43.985229234s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254079065 (211h38m43.985346696s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254257481 (211h38m43.985525111s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254408585 (211h38m43.985676204s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254523081 (211h38m43.985790721s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254695415 (211h38m43.985963045s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254849924 (211h38m43.986117554s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.254965993 (211h38m43.986233613s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255073975 (211h38m43.986341606s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255232403 (211h38m43.986500033s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255467977 (211h38m43.986735607s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255719861 (211h38m43.986987481s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.255961866 (211h38m43.987229496s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Uploading piece... Aug 17 13:05:03.323589897 (211h38m44.054857527s) [upload=220194]: Next returned nil segment Aug 17 13:05:03.323600959 (211h38m44.054868579s) [upload=220194]: Waiting for error group managing segments... Aug 17 13:05:03.351486953 (211h38m44.082754583s) [noise=true , address=45.91.116.10:4108 , piece_id=UJP6ESGMCDM52MG7CPCAGM6HFURCNFMC3MP6PF6VLTS2YKDH3ZWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.356136245 (211h38m44.087403865s) [noise=true , address=77.90.31.18:4838 , piece_id=ABIHECHOE7TXF7ZNEP45CMCAHWHRA4O6L4WLWXI3PRJHHDRU2OSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.359798501 (211h38m44.091066121s) [noise=true , address=77.90.31.16:4758 , piece_id=ZYUOJ663TRGB67BXC7LUGQRQGGSU56LD4TQLYR6E2R7CVQNWY2SQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.368712994 (211h38m44.099980613s) [noise=true , address=77.90.31.12:4624 , piece_id=QEDSYM53SGPAMLX7LNF4SGGUOHXX6VGVXBAYYY7HVG3D76HMYA6Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.451425619 (211h38m44.18269327s) [noise=true , address=77.90.31.13:4640 , piece_id=XLNBM6CDBXTG5ZRZFNBHAXV4CNLHBS4MM4MI5ATRINW6NYON4XBQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.488338257 (211h38m44.219605887s) [noise=true , address=45.91.116.6:4993 , piece_id=HGNWD6LQU7RF5JUT2DI5MFBBGIFN2QVTXBPAEIHINYQH5RSJMQ4Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.533602960 (211h38m44.26487062s) [noise=true , address=79.127.201.215:4349 , piece_id=TLF6ZGUDNPGNHU65NSB3MCVHELD5SRALOX5A5EYEMWZEA7CUNRZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.706703573 (211h38m44.437971194s) [noise=true , address=45.91.116.9:4284 , piece_id=7XM6UXEX7XCE53VUSVE5FVHOSLHZ3O7HINNBYIIPREWHZFTJBXQA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.845738394 (211h38m44.577006013s) [noise=true , address=77.90.31.11:4565 , piece_id=X5UWETN3VWNI5QELVHRWFIBVXX4XSBJ6PZYS3J3XLSKFKHK46KZA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:05:03.854673865 (211h38m44.585941515s) [noise=true , address=66.23.193.175:4562 , piece_id=K3MUJ7MX4WNQ6BIECEHIZJYEGB27VNUMF4KAEIRTQLMZ72BTCHGA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: <nil> Aug 17 13:06:03.213670588 (211h39m43.944938208s) [noise=true , address=104.36.82.12:4169 , piece_id=OR54CFSBKIN5VDRRPZZODSF4CFX75EUIC2GO25MW2B4UDY2NXM2Q, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213738616 (211h39m43.945006266s) [noise=true , address=104.36.82.60:4363 , piece_id=NJYMMSW7XMUSJVDILBJX3WEINUSHN2O5CIW2QTIUNI6SUJ2PLFOA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7, address:104.36.82.60:4363): protocol: write tcp 5.161.89.252:44086->104.36.82.60:4363: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213807617 (211h39m43.945075236s) [noise=true , address=104.36.82.44:4871 , piece_id=LRT2Z4WZ45CLC2QCR4WDEGQBER7SPXUPID3IAPQEQRBLIG3I5ZEQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213844345 (211h39m43.945111965s) [noise=true , address=104.36.82.11:4138 , piece_id=NMWQB3XGW7KJNHYI427JFDLXWLSGTFOG7NSCESBXAXDLIGKIWGWA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213873840 (211h39m43.945141471s) [noise=true , address=104.36.82.9:4025 , piece_id=GX7UXJEE6TGXCE4VASFCTN7U4TU7UPFQ7XVEA5OFFRR7XRNKROIQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 5.161.89.252:16954->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213895070 (211h39m43.945162691s) [noise=true , address=104.36.82.17:4358 , piece_id=MTXF3E6B7UXC24I66GQF6ACWHVEYFHWBQOTGNZN3RXTVNFGITZIA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: rpc: dial tcp 104.36.82.17:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213912293 (211h39m43.945179912s) [noise=true , address=104.36.82.23:4462 , piece_id=NLJSKNDBAP73B4SPXIGBNYDVMACLZIQZ7ULDJE7YBAKKX53X5ZSQ, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: rpc: dial tcp 104.36.82.23:4462: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213916270 (211h39m43.94518389s) [noise=true , address=104.36.82.10:4101 , piece_id=3KVWATFDFOU6RQVO7UDOQGYKOFG4VK77LURI5DPEWIGE7F4RYHAA, seg_pos={0 0}, upload=220194]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 13:06:03.213938842 (211h39m43.945206472s) [seg_pos={0 0}, upload=220194]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 13:06:03.239464369 (211h39m43.970731999s) [upload=220194]: Done uploading segments... Aug 17 13:06:03.239484097 (211h39m43.970751716s) [upload=220194]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 778.688488ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 537.342238ms FAIL (2025-08-17 13:35:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 13:00:24.431681640 (211h34m5.162949261s) [upload=220113]: Starting upload Aug 17 13:00:24.431716026 (211h34m5.162983646s) [upload=220113]: Done starting upload Aug 17 13:00:24.431759107 (211h34m5.163026727s) [upload=220113]: Uploading segments... Aug 17 13:00:24.431790736 (211h34m5.163058366s) [upload=220113]: Got next segment. Inline: true Aug 17 13:00:24.431795495 (211h34m5.163063115s) [upload=220113]: Waiting for error group managing segments... Aug 17 13:00:24.431947481 (211h34m5.163215111s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 13:00:24.431953533 (211h34m5.163221163s) [upload=220113]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 13:00:24.431956157 (211h34m5.163223787s) [upload=220113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 13:00:24.434700137 (211h34m5.165967787s) [upload=220113]: Done uploading segments... Aug 17 13:00:24.434709414 (211h34m5.165977034s) [upload=220113]: Upload finished. err: metaclient: Too Many Requests Duration:* 537.35882ms core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)