Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 236.863328ms FAIL (2025-08-27 21:35:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 27 21:01:41.377970271 (3h23m4.607455648s) [upload=3507]: Starting upload Aug 27 21:01:41.378099755 (3h23m4.607585132s) [upload=3507]: Done starting upload Aug 27 21:01:41.378244187 (3h23m4.607729564s) [upload=3507]: Uploading segments... Aug 27 21:01:41.378310020 (3h23m4.607795397s) [upload=3507]: Got next segment. Inline: true Aug 27 21:01:41.378315851 (3h23m4.607801238s) [upload=3507]: Waiting for error group managing segments... Aug 27 21:01:41.378361376 (3h23m4.607846764s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:41.378366396 (3h23m4.607851773s) [upload=3507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:01:41.378370033 (3h23m4.607855409s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:01:53.431192035 (3h23m16.660677392s) [upload=3507]: Done uploading segments... Aug 27 21:01:53.431206482 (3h23m16.660691839s) [upload=3507]: Upload finished. err: <nil> Duration:* 236.88548ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 420.741837ms FAIL (2025-08-27 21:35:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 27 21:01:41.377970271 (3h23m4.607455648s) [upload=3507]: Starting upload Aug 27 21:01:41.378099755 (3h23m4.607585132s) [upload=3507]: Done starting upload Aug 27 21:01:41.378244187 (3h23m4.607729564s) [upload=3507]: Uploading segments... Aug 27 21:01:41.378310020 (3h23m4.607795397s) [upload=3507]: Got next segment. Inline: true Aug 27 21:01:41.378315851 (3h23m4.607801238s) [upload=3507]: Waiting for error group managing segments... Aug 27 21:01:41.378361376 (3h23m4.607846764s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:41.378366396 (3h23m4.607851773s) [upload=3507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:01:41.378370033 (3h23m4.607855409s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:01:53.431192035 (3h23m16.660677392s) [upload=3507]: Done uploading segments... Aug 27 21:01:53.431206482 (3h23m16.660691839s) [upload=3507]: Upload finished. err: <nil> Duration:* 420.767887ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 454.073729ms FAIL (2025-08-27 21:35:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 454.112623ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.770191919s FAIL (2025-08-27 21:35:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.770217577s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 353.80901ms FAIL (2025-08-27 21:35:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 353.83597ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 255.660686ms FAIL (2025-08-27 21:34:59 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: accounting backend: Redis eval failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:07.426688152 (3h23m30.656173499s) [upload=3515]: Starting upload Aug 27 21:02:07.426735873 (3h23m30.65622122s) [upload=3515]: Done starting upload Aug 27 21:02:07.426790635 (3h23m30.656275992s) [upload=3515]: Uploading segments... Aug 27 21:02:07.426837303 (3h23m30.65632265s) [upload=3515]: Got next segment. Inline: true Aug 27 21:02:07.426841050 (3h23m30.656326387s) [upload=3515]: Waiting for error group managing segments... Aug 27 21:02:07.426872439 (3h23m30.656357786s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:07.426875605 (3h23m30.656360952s) [upload=3515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:02:07.426877649 (3h23m30.656362995s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:20.276735982 (3h23m43.506221359s) [upload=3515]: Done uploading segments... Aug 27 21:02:20.276756050 (3h23m43.506241427s) [upload=3515]: Upload finished. err: <nil> Duration:* 255.681726ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 153.820801ms FAIL (2025-08-27 21:34:42 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: accounting backend: Redis eval failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:07.426688152 (3h23m30.656173499s) [upload=3515]: Starting upload Aug 27 21:02:07.426735873 (3h23m30.65622122s) [upload=3515]: Done starting upload Aug 27 21:02:07.426790635 (3h23m30.656275992s) [upload=3515]: Uploading segments... Aug 27 21:02:07.426837303 (3h23m30.65632265s) [upload=3515]: Got next segment. Inline: true Aug 27 21:02:07.426841050 (3h23m30.656326387s) [upload=3515]: Waiting for error group managing segments... Aug 27 21:02:07.426872439 (3h23m30.656357786s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:07.426875605 (3h23m30.656360952s) [upload=3515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:02:07.426877649 (3h23m30.656362995s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:20.276735982 (3h23m43.506221359s) [upload=3515]: Done uploading segments... Aug 27 21:02:20.276756050 (3h23m43.506241427s) [upload=3515]: Upload finished. err: <nil> Duration:* 153.841882ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 406.705131ms FAIL (2025-08-27 21:34:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 27 21:01:41.377970271 (3h23m4.607455648s) [upload=3507]: Starting upload Aug 27 21:01:41.378099755 (3h23m4.607585132s) [upload=3507]: Done starting upload Aug 27 21:01:41.378244187 (3h23m4.607729564s) [upload=3507]: Uploading segments... Aug 27 21:01:41.378310020 (3h23m4.607795397s) [upload=3507]: Got next segment. Inline: true Aug 27 21:01:41.378315851 (3h23m4.607801238s) [upload=3507]: Waiting for error group managing segments... Aug 27 21:01:41.378361376 (3h23m4.607846764s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:41.378366396 (3h23m4.607851773s) [upload=3507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:01:41.378370033 (3h23m4.607855409s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:01:53.431192035 (3h23m16.660677392s) [upload=3507]: Done uploading segments... Aug 27 21:01:53.431206482 (3h23m16.660691839s) [upload=3507]: Upload finished. err: <nil> Duration:* 406.728033ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 424.121967ms FAIL (2025-08-27 21:34:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 424.141094ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 453.454494ms FAIL (2025-08-27 21:34:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 27 21:01:36.226560503 (3h22m59.456045849s) [upload=3504]: Starting upload Aug 27 21:01:36.226622129 (3h22m59.456107476s) [upload=3504]: Done starting upload Aug 27 21:01:36.226666592 (3h22m59.456151949s) [upload=3504]: Uploading segments... Aug 27 21:01:36.226834699 (3h22m59.456320045s) [upload=3504]: Got next segment. Inline: false Aug 27 21:01:36.226865156 (3h22m59.456350502s) [upload=3504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:36.226869904 (3h22m59.456355252s) [upload=3504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:36.212957093 (3h23m59.44244245s) [upload=3504]: Next segment err: context deadline exceeded Aug 27 21:02:36.213033316 (3h23m59.442518673s) [upload=3504]: Done uploading segments... Aug 27 21:02:36.213048534 (3h23m59.442533891s) [upload=3504]: Upload finished. err: context deadline exceeded Duration: 453.47955ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.880877446s FAIL (2025-08-27 21:34:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.880894418s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 425.141477ms FAIL (2025-08-27 21:34:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 425.162455ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 498.457995ms FAIL (2025-08-27 21:34:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 498.481168ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 397.76037ms FAIL (2025-08-27 21:34:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 397.787512ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 620.546162ms FAIL (2025-08-27 21:33:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 620.574897ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 412.040447ms FAIL (2025-08-27 21:33:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 27 21:01:36.226560503 (3h22m59.456045849s) [upload=3504]: Starting upload Aug 27 21:01:36.226622129 (3h22m59.456107476s) [upload=3504]: Done starting upload Aug 27 21:01:36.226666592 (3h22m59.456151949s) [upload=3504]: Uploading segments... Aug 27 21:01:36.226834699 (3h22m59.456320045s) [upload=3504]: Got next segment. Inline: false Aug 27 21:01:36.226865156 (3h22m59.456350502s) [upload=3504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:36.226869904 (3h22m59.456355252s) [upload=3504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:36.212957093 (3h23m59.44244245s) [upload=3504]: Next segment err: context deadline exceeded Aug 27 21:02:36.213033316 (3h23m59.442518673s) [upload=3504]: Done uploading segments... Aug 27 21:02:36.213048534 (3h23m59.442533891s) [upload=3504]: Upload finished. err: context deadline exceeded Duration: 412.061278ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 454.186263ms FAIL (2025-08-27 21:33:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 27 21:01:41.377970271 (3h23m4.607455648s) [upload=3507]: Starting upload Aug 27 21:01:41.378099755 (3h23m4.607585132s) [upload=3507]: Done starting upload Aug 27 21:01:41.378244187 (3h23m4.607729564s) [upload=3507]: Uploading segments... Aug 27 21:01:41.378310020 (3h23m4.607795397s) [upload=3507]: Got next segment. Inline: true Aug 27 21:01:41.378315851 (3h23m4.607801238s) [upload=3507]: Waiting for error group managing segments... Aug 27 21:01:41.378361376 (3h23m4.607846764s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:41.378366396 (3h23m4.607851773s) [upload=3507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:01:41.378370033 (3h23m4.607855409s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:01:53.431192035 (3h23m16.660677392s) [upload=3507]: Done uploading segments... Aug 27 21:01:53.431206482 (3h23m16.660691839s) [upload=3507]: Upload finished. err: <nil> Duration:* 454.213755ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 205.643908ms FAIL (2025-08-27 21:33:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: accounting backend: Redis eval failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:07.426688152 (3h23m30.656173499s) [upload=3515]: Starting upload Aug 27 21:02:07.426735873 (3h23m30.65622122s) [upload=3515]: Done starting upload Aug 27 21:02:07.426790635 (3h23m30.656275992s) [upload=3515]: Uploading segments... Aug 27 21:02:07.426837303 (3h23m30.65632265s) [upload=3515]: Got next segment. Inline: true Aug 27 21:02:07.426841050 (3h23m30.656326387s) [upload=3515]: Waiting for error group managing segments... Aug 27 21:02:07.426872439 (3h23m30.656357786s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:07.426875605 (3h23m30.656360952s) [upload=3515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:02:07.426877649 (3h23m30.656362995s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:20.276735982 (3h23m43.506221359s) [upload=3515]: Done uploading segments... Aug 27 21:02:20.276756050 (3h23m43.506241427s) [upload=3515]: Upload finished. err: <nil> Duration:* 205.669226ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.800195748s FAIL (2025-08-27 21:33:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.800222068s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 588.299213ms FAIL (2025-08-27 21:33:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 588.423729ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 401.265495ms FAIL (2025-08-27 21:33:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 401.293186ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.56430957s FAIL (2025-08-27 21:33:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.564334898s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 474.781392ms FAIL (2025-08-27 21:32:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 474.804235ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 176.707451ms FAIL (2025-08-27 21:32:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 176.734493ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 394.657513ms FAIL (2025-08-27 21:32:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 394.67643ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 215.759423ms FAIL (2025-08-27 21:32:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 27 21:01:41.377970271 (3h23m4.607455648s) [upload=3507]: Starting upload Aug 27 21:01:41.378099755 (3h23m4.607585132s) [upload=3507]: Done starting upload Aug 27 21:01:41.378244187 (3h23m4.607729564s) [upload=3507]: Uploading segments... Aug 27 21:01:41.378310020 (3h23m4.607795397s) [upload=3507]: Got next segment. Inline: true Aug 27 21:01:41.378315851 (3h23m4.607801238s) [upload=3507]: Waiting for error group managing segments... Aug 27 21:01:41.378361376 (3h23m4.607846764s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:41.378366396 (3h23m4.607851773s) [upload=3507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:01:41.378370033 (3h23m4.607855409s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:01:53.431192035 (3h23m16.660677392s) [upload=3507]: Done uploading segments... Aug 27 21:01:53.431206482 (3h23m16.660691839s) [upload=3507]: Upload finished. err: <nil> Duration:* 215.784279ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 638.292301ms FAIL (2025-08-27 21:32:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 638.312629ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 688.696389ms FAIL (2025-08-27 21:32:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 27 21:01:36.226560503 (3h22m59.456045849s) [upload=3504]: Starting upload Aug 27 21:01:36.226622129 (3h22m59.456107476s) [upload=3504]: Done starting upload Aug 27 21:01:36.226666592 (3h22m59.456151949s) [upload=3504]: Uploading segments... Aug 27 21:01:36.226834699 (3h22m59.456320045s) [upload=3504]: Got next segment. Inline: false Aug 27 21:01:36.226865156 (3h22m59.456350502s) [upload=3504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:36.226869904 (3h22m59.456355252s) [upload=3504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:36.212957093 (3h23m59.44244245s) [upload=3504]: Next segment err: context deadline exceeded Aug 27 21:02:36.213033316 (3h23m59.442518673s) [upload=3504]: Done uploading segments... Aug 27 21:02:36.213048534 (3h23m59.442533891s) [upload=3504]: Upload finished. err: context deadline exceeded Duration: 688.726176ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 210.673334ms FAIL (2025-08-27 21:32:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: accounting backend: Redis eval failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:07.426688152 (3h23m30.656173499s) [upload=3515]: Starting upload Aug 27 21:02:07.426735873 (3h23m30.65622122s) [upload=3515]: Done starting upload Aug 27 21:02:07.426790635 (3h23m30.656275992s) [upload=3515]: Uploading segments... Aug 27 21:02:07.426837303 (3h23m30.65632265s) [upload=3515]: Got next segment. Inline: true Aug 27 21:02:07.426841050 (3h23m30.656326387s) [upload=3515]: Waiting for error group managing segments... Aug 27 21:02:07.426872439 (3h23m30.656357786s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:07.426875605 (3h23m30.656360952s) [upload=3515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:02:07.426877649 (3h23m30.656362995s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:20.276735982 (3h23m43.506221359s) [upload=3515]: Done uploading segments... Aug 27 21:02:20.276756050 (3h23m43.506241427s) [upload=3515]: Upload finished. err: <nil> Duration:* 210.705235ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.............X.............................................] duration: 495.576444ms FAIL (2025-08-27 21:32:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 495.596581ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.890396644s FAIL (2025-08-27 21:32:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.8904163s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 205.489761ms FAIL (2025-08-27 21:32:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 27 21:01:41.377970271 (3h23m4.607455648s) [upload=3507]: Starting upload Aug 27 21:01:41.378099755 (3h23m4.607585132s) [upload=3507]: Done starting upload Aug 27 21:01:41.378244187 (3h23m4.607729564s) [upload=3507]: Uploading segments... Aug 27 21:01:41.378310020 (3h23m4.607795397s) [upload=3507]: Got next segment. Inline: true Aug 27 21:01:41.378315851 (3h23m4.607801238s) [upload=3507]: Waiting for error group managing segments... Aug 27 21:01:41.378361376 (3h23m4.607846764s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:41.378366396 (3h23m4.607851773s) [upload=3507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:01:41.378370033 (3h23m4.607855409s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:01:53.431192035 (3h23m16.660677392s) [upload=3507]: Done uploading segments... Aug 27 21:01:53.431206482 (3h23m16.660691839s) [upload=3507]: Upload finished. err: <nil> Duration:* 205.51051ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 663.446055ms FAIL (2025-08-27 21:32:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 27 21:01:36.226560503 (3h22m59.456045849s) [upload=3504]: Starting upload Aug 27 21:01:36.226622129 (3h22m59.456107476s) [upload=3504]: Done starting upload Aug 27 21:01:36.226666592 (3h22m59.456151949s) [upload=3504]: Uploading segments... Aug 27 21:01:36.226834699 (3h22m59.456320045s) [upload=3504]: Got next segment. Inline: false Aug 27 21:01:36.226865156 (3h22m59.456350502s) [upload=3504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:36.226869904 (3h22m59.456355252s) [upload=3504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:36.212957093 (3h23m59.44244245s) [upload=3504]: Next segment err: context deadline exceeded Aug 27 21:02:36.213033316 (3h23m59.442518673s) [upload=3504]: Done uploading segments... Aug 27 21:02:36.213048534 (3h23m59.442533891s) [upload=3504]: Upload finished. err: context deadline exceeded Duration: 663.470481ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 236.950312ms FAIL (2025-08-27 21:31:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 236.982031ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 602.058719ms FAIL (2025-08-27 21:31:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 602.078646ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.............X............................................] duration: 551.845227ms FAIL (2025-08-27 21:31:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 551.868831ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.849082825s FAIL (2025-08-27 21:31:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.849108623s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 248.056006ms FAIL (2025-08-27 21:31:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: accounting backend: Redis eval failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:07.426688152 (3h23m30.656173499s) [upload=3515]: Starting upload Aug 27 21:02:07.426735873 (3h23m30.65622122s) [upload=3515]: Done starting upload Aug 27 21:02:07.426790635 (3h23m30.656275992s) [upload=3515]: Uploading segments... Aug 27 21:02:07.426837303 (3h23m30.65632265s) [upload=3515]: Got next segment. Inline: true Aug 27 21:02:07.426841050 (3h23m30.656326387s) [upload=3515]: Waiting for error group managing segments... Aug 27 21:02:07.426872439 (3h23m30.656357786s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:07.426875605 (3h23m30.656360952s) [upload=3515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:02:07.426877649 (3h23m30.656362995s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:20.276735982 (3h23m43.506221359s) [upload=3515]: Done uploading segments... Aug 27 21:02:20.276756050 (3h23m43.506241427s) [upload=3515]: Upload finished. err: <nil> Duration:* 248.074961ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 494.667684ms FAIL (2025-08-27 21:31:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 494.69279ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 391.109528ms FAIL (2025-08-27 21:30:59 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 27 21:01:41.377970271 (3h23m4.607455648s) [upload=3507]: Starting upload Aug 27 21:01:41.378099755 (3h23m4.607585132s) [upload=3507]: Done starting upload Aug 27 21:01:41.378244187 (3h23m4.607729564s) [upload=3507]: Uploading segments... Aug 27 21:01:41.378310020 (3h23m4.607795397s) [upload=3507]: Got next segment. Inline: true Aug 27 21:01:41.378315851 (3h23m4.607801238s) [upload=3507]: Waiting for error group managing segments... Aug 27 21:01:41.378361376 (3h23m4.607846764s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:41.378366396 (3h23m4.607851773s) [upload=3507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:01:41.378370033 (3h23m4.607855409s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:01:53.431192035 (3h23m16.660677392s) [upload=3507]: Done uploading segments... Aug 27 21:01:53.431206482 (3h23m16.660691839s) [upload=3507]: Upload finished. err: <nil> Duration:* 391.139624ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 485.451343ms FAIL (2025-08-27 21:30:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 27 21:01:36.226560503 (3h22m59.456045849s) [upload=3504]: Starting upload Aug 27 21:01:36.226622129 (3h22m59.456107476s) [upload=3504]: Done starting upload Aug 27 21:01:36.226666592 (3h22m59.456151949s) [upload=3504]: Uploading segments... Aug 27 21:01:36.226834699 (3h22m59.456320045s) [upload=3504]: Got next segment. Inline: false Aug 27 21:01:36.226865156 (3h22m59.456350502s) [upload=3504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:36.226869904 (3h22m59.456355252s) [upload=3504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:36.212957093 (3h23m59.44244245s) [upload=3504]: Next segment err: context deadline exceeded Aug 27 21:02:36.213033316 (3h23m59.442518673s) [upload=3504]: Done uploading segments... Aug 27 21:02:36.213048534 (3h23m59.442533891s) [upload=3504]: Upload finished. err: context deadline exceeded Duration: 485.469137ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.825837653s FAIL (2025-08-27 21:30:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.825878841s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 424.083726ms FAIL (2025-08-27 21:30:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 424.116268ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.............X...........................................] duration: 749.851889ms FAIL (2025-08-27 21:30:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 749.878919ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 560.101751ms FAIL (2025-08-27 21:30:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 27 21:01:36.226560503 (3h22m59.456045849s) [upload=3504]: Starting upload Aug 27 21:01:36.226622129 (3h22m59.456107476s) [upload=3504]: Done starting upload Aug 27 21:01:36.226666592 (3h22m59.456151949s) [upload=3504]: Uploading segments... Aug 27 21:01:36.226834699 (3h22m59.456320045s) [upload=3504]: Got next segment. Inline: false Aug 27 21:01:36.226865156 (3h22m59.456350502s) [upload=3504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:36.226869904 (3h22m59.456355252s) [upload=3504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:36.212957093 (3h23m59.44244245s) [upload=3504]: Next segment err: context deadline exceeded Aug 27 21:02:36.213033316 (3h23m59.442518673s) [upload=3504]: Done uploading segments... Aug 27 21:02:36.213048534 (3h23m59.442533891s) [upload=3504]: Upload finished. err: context deadline exceeded Duration: 560.126477ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 489.89343ms FAIL (2025-08-27 21:30:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 489.91989ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.............X..........................................] duration: 613.269143ms FAIL (2025-08-27 21:30:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 613.291004ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 134.774695ms FAIL (2025-08-27 21:30:18 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: accounting backend: Redis eval failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:07.426688152 (3h23m30.656173499s) [upload=3515]: Starting upload Aug 27 21:02:07.426735873 (3h23m30.65622122s) [upload=3515]: Done starting upload Aug 27 21:02:07.426790635 (3h23m30.656275992s) [upload=3515]: Uploading segments... Aug 27 21:02:07.426837303 (3h23m30.65632265s) [upload=3515]: Got next segment. Inline: true Aug 27 21:02:07.426841050 (3h23m30.656326387s) [upload=3515]: Waiting for error group managing segments... Aug 27 21:02:07.426872439 (3h23m30.656357786s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:07.426875605 (3h23m30.656360952s) [upload=3515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:02:07.426877649 (3h23m30.656362995s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:20.276735982 (3h23m43.506221359s) [upload=3515]: Done uploading segments... Aug 27 21:02:20.276756050 (3h23m43.506241427s) [upload=3515]: Upload finished. err: <nil> Duration:* 134.797859ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 740.414935ms FAIL (2025-08-27 21:30:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 740.479318ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 193.658761ms FAIL (2025-08-27 21:30:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 193.680192ms core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)