Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.622420708s FAIL (2025-08-27 21:29:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.622446588s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.............X.........................................] duration: 775.005355ms FAIL (2025-08-27 21:29:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 775.036604ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 206.958894ms FAIL (2025-08-27 21:29:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 27 21:01:41.377970271 (3h23m4.607455648s) [upload=3507]: Starting upload Aug 27 21:01:41.378099755 (3h23m4.607585132s) [upload=3507]: Done starting upload Aug 27 21:01:41.378244187 (3h23m4.607729564s) [upload=3507]: Uploading segments... Aug 27 21:01:41.378310020 (3h23m4.607795397s) [upload=3507]: Got next segment. Inline: true Aug 27 21:01:41.378315851 (3h23m4.607801238s) [upload=3507]: Waiting for error group managing segments... Aug 27 21:01:41.378361376 (3h23m4.607846764s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:41.378366396 (3h23m4.607851773s) [upload=3507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:01:41.378370033 (3h23m4.607855409s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:01:53.431192035 (3h23m16.660677392s) [upload=3507]: Done uploading segments... Aug 27 21:01:53.431206482 (3h23m16.660691839s) [upload=3507]: Upload finished. err: <nil> Duration:* 206.976667ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 445.906601ms FAIL (2025-08-27 21:29:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 27 21:01:36.226560503 (3h22m59.456045849s) [upload=3504]: Starting upload Aug 27 21:01:36.226622129 (3h22m59.456107476s) [upload=3504]: Done starting upload Aug 27 21:01:36.226666592 (3h22m59.456151949s) [upload=3504]: Uploading segments... Aug 27 21:01:36.226834699 (3h22m59.456320045s) [upload=3504]: Got next segment. Inline: false Aug 27 21:01:36.226865156 (3h22m59.456350502s) [upload=3504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:36.226869904 (3h22m59.456355252s) [upload=3504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:36.212957093 (3h23m59.44244245s) [upload=3504]: Next segment err: context deadline exceeded Aug 27 21:02:36.213033316 (3h23m59.442518673s) [upload=3504]: Done uploading segments... Aug 27 21:02:36.213048534 (3h23m59.442533891s) [upload=3504]: Upload finished. err: context deadline exceeded Duration: 445.929946ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 306.753679ms FAIL (2025-08-27 21:29:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 306.776322ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 508.911081ms FAIL (2025-08-27 21:29:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 508.934425ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 410.550082ms FAIL (2025-08-27 21:29:39 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: accounting backend: Redis eval failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:07.426688152 (3h23m30.656173499s) [upload=3515]: Starting upload Aug 27 21:02:07.426735873 (3h23m30.65622122s) [upload=3515]: Done starting upload Aug 27 21:02:07.426790635 (3h23m30.656275992s) [upload=3515]: Uploading segments... Aug 27 21:02:07.426837303 (3h23m30.65632265s) [upload=3515]: Got next segment. Inline: true Aug 27 21:02:07.426841050 (3h23m30.656326387s) [upload=3515]: Waiting for error group managing segments... Aug 27 21:02:07.426872439 (3h23m30.656357786s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:07.426875605 (3h23m30.656360952s) [upload=3515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:02:07.426877649 (3h23m30.656362995s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:20.276735982 (3h23m43.506221359s) [upload=3515]: Done uploading segments... Aug 27 21:02:20.276756050 (3h23m43.506241427s) [upload=3515]: Upload finished. err: <nil> Duration:* 410.572897ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.715864243s FAIL (2025-08-27 21:29:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.715890351s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 280.683812ms FAIL (2025-08-27 21:29:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 280.710642ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 403.795937ms FAIL (2025-08-27 21:29:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 403.815824ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 206.551557ms FAIL (2025-08-27 21:28:57 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 27 21:01:41.377970271 (3h23m4.607455648s) [upload=3507]: Starting upload Aug 27 21:01:41.378099755 (3h23m4.607585132s) [upload=3507]: Done starting upload Aug 27 21:01:41.378244187 (3h23m4.607729564s) [upload=3507]: Uploading segments... Aug 27 21:01:41.378310020 (3h23m4.607795397s) [upload=3507]: Got next segment. Inline: true Aug 27 21:01:41.378315851 (3h23m4.607801238s) [upload=3507]: Waiting for error group managing segments... Aug 27 21:01:41.378361376 (3h23m4.607846764s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:41.378366396 (3h23m4.607851773s) [upload=3507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:01:41.378370033 (3h23m4.607855409s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:01:53.431192035 (3h23m16.660677392s) [upload=3507]: Done uploading segments... Aug 27 21:01:53.431206482 (3h23m16.660691839s) [upload=3507]: Upload finished. err: <nil> Duration:* 206.572107ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 192.628863ms FAIL (2025-08-27 21:28:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: accounting backend: Redis eval failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:07.426688152 (3h23m30.656173499s) [upload=3515]: Starting upload Aug 27 21:02:07.426735873 (3h23m30.65622122s) [upload=3515]: Done starting upload Aug 27 21:02:07.426790635 (3h23m30.656275992s) [upload=3515]: Uploading segments... Aug 27 21:02:07.426837303 (3h23m30.65632265s) [upload=3515]: Got next segment. Inline: true Aug 27 21:02:07.426841050 (3h23m30.656326387s) [upload=3515]: Waiting for error group managing segments... Aug 27 21:02:07.426872439 (3h23m30.656357786s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:07.426875605 (3h23m30.656360952s) [upload=3515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:02:07.426877649 (3h23m30.656362995s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:20.276735982 (3h23m43.506221359s) [upload=3515]: Done uploading segments... Aug 27 21:02:20.276756050 (3h23m43.506241427s) [upload=3515]: Upload finished. err: <nil> Duration:* 192.651898ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.864737867s FAIL (2025-08-27 21:28:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.864759177s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.............X........................................] duration: 436.898677ms FAIL (2025-08-27 21:28:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 436.92216ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 719.242402ms FAIL (2025-08-27 21:28:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 27 21:01:36.226560503 (3h22m59.456045849s) [upload=3504]: Starting upload Aug 27 21:01:36.226622129 (3h22m59.456107476s) [upload=3504]: Done starting upload Aug 27 21:01:36.226666592 (3h22m59.456151949s) [upload=3504]: Uploading segments... Aug 27 21:01:36.226834699 (3h22m59.456320045s) [upload=3504]: Got next segment. Inline: false Aug 27 21:01:36.226865156 (3h22m59.456350502s) [upload=3504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:36.226869904 (3h22m59.456355252s) [upload=3504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:36.212957093 (3h23m59.44244245s) [upload=3504]: Next segment err: context deadline exceeded Aug 27 21:02:36.213033316 (3h23m59.442518673s) [upload=3504]: Done uploading segments... Aug 27 21:02:36.213048534 (3h23m59.442533891s) [upload=3504]: Upload finished. err: context deadline exceeded Duration: 719.29464ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 410.655301ms FAIL (2025-08-27 21:28:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 410.675468ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.............X.......................................] duration: 711.339903ms FAIL (2025-08-27 21:28:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 711.366834ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 581.919018ms FAIL (2025-08-27 21:28:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 581.940418ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 213.400824ms FAIL (2025-08-27 21:28:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 213.430932ms core uplink us1
checkers.HTTP{"GET" "https://www.tardigrade.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.tardigrade_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 40.294911ms remote: "tcp:34.120.119.150:443" response headers: Referrer-Policy=no-referrer Content-Length=332 Date=Wed, 27 Aug 2025 21:14:56 GMT Alt-Svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Content-Type=text/html; charset=UTF-8 target: https://www.tardigrade.io/ FAIL (2025-08-27 21:27:57 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>502 Server Error</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Server Error</h1>\n<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n<h2></h2>\n</body></html>\n" Duration: 40.318856ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 482.657226ms FAIL (2025-08-27 21:27:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 27 21:01:36.226560503 (3h22m59.456045849s) [upload=3504]: Starting upload Aug 27 21:01:36.226622129 (3h22m59.456107476s) [upload=3504]: Done starting upload Aug 27 21:01:36.226666592 (3h22m59.456151949s) [upload=3504]: Uploading segments... Aug 27 21:01:36.226834699 (3h22m59.456320045s) [upload=3504]: Got next segment. Inline: false Aug 27 21:01:36.226865156 (3h22m59.456350502s) [upload=3504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:36.226869904 (3h22m59.456355252s) [upload=3504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:36.212957093 (3h23m59.44244245s) [upload=3504]: Next segment err: context deadline exceeded Aug 27 21:02:36.213033316 (3h23m59.442518673s) [upload=3504]: Done uploading segments... Aug 27 21:02:36.213048534 (3h23m59.442533891s) [upload=3504]: Upload finished. err: context deadline exceeded Duration: 482.678776ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 178.442154ms FAIL (2025-08-27 21:27:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 27 21:01:41.377970271 (3h23m4.607455648s) [upload=3507]: Starting upload Aug 27 21:01:41.378099755 (3h23m4.607585132s) [upload=3507]: Done starting upload Aug 27 21:01:41.378244187 (3h23m4.607729564s) [upload=3507]: Uploading segments... Aug 27 21:01:41.378310020 (3h23m4.607795397s) [upload=3507]: Got next segment. Inline: true Aug 27 21:01:41.378315851 (3h23m4.607801238s) [upload=3507]: Waiting for error group managing segments... Aug 27 21:01:41.378361376 (3h23m4.607846764s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:41.378366396 (3h23m4.607851773s) [upload=3507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:01:41.378370033 (3h23m4.607855409s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:01:53.431192035 (3h23m16.660677392s) [upload=3507]: Done uploading segments... Aug 27 21:01:53.431206482 (3h23m16.660691839s) [upload=3507]: Upload finished. err: <nil> Duration:* 178.466559ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 493.627977ms FAIL (2025-08-27 21:27:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: accounting backend: Redis eval failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:07.426688152 (3h23m30.656173499s) [upload=3515]: Starting upload Aug 27 21:02:07.426735873 (3h23m30.65622122s) [upload=3515]: Done starting upload Aug 27 21:02:07.426790635 (3h23m30.656275992s) [upload=3515]: Uploading segments... Aug 27 21:02:07.426837303 (3h23m30.65632265s) [upload=3515]: Got next segment. Inline: true Aug 27 21:02:07.426841050 (3h23m30.656326387s) [upload=3515]: Waiting for error group managing segments... Aug 27 21:02:07.426872439 (3h23m30.656357786s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:07.426875605 (3h23m30.656360952s) [upload=3515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:02:07.426877649 (3h23m30.656362995s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:20.276735982 (3h23m43.506221359s) [upload=3515]: Done uploading segments... Aug 27 21:02:20.276756050 (3h23m43.506241427s) [upload=3515]: Upload finished. err: <nil> Duration:* 493.652324ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.............X......................................] duration: 717.863488ms FAIL (2025-08-27 21:27:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 717.887122ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.631631681s FAIL (2025-08-27 21:27:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.631664172s core uplink us1
checkers.HTTP{"GET" "https://storj.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 41.987196ms remote: "tcp:34.120.119.150:443" response headers: Alt-Svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Content-Type=text/html; charset=UTF-8 Referrer-Policy=no-referrer Content-Length=332 Date=Wed, 27 Aug 2025 21:17:16 GMT target: https://storj.io/ FAIL (2025-08-27 21:27:35 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>502 Server Error</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Server Error</h1>\n<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n<h2></h2>\n</body></html>\n" Duration: 42.011481ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 199.477039ms FAIL (2025-08-27 21:27:34 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 27 21:01:41.377970271 (3h23m4.607455648s) [upload=3507]: Starting upload Aug 27 21:01:41.378099755 (3h23m4.607585132s) [upload=3507]: Done starting upload Aug 27 21:01:41.378244187 (3h23m4.607729564s) [upload=3507]: Uploading segments... Aug 27 21:01:41.378310020 (3h23m4.607795397s) [upload=3507]: Got next segment. Inline: true Aug 27 21:01:41.378315851 (3h23m4.607801238s) [upload=3507]: Waiting for error group managing segments... Aug 27 21:01:41.378361376 (3h23m4.607846764s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:41.378366396 (3h23m4.607851773s) [upload=3507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:01:41.378370033 (3h23m4.607855409s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:01:53.431192035 (3h23m16.660677392s) [upload=3507]: Done uploading segments... Aug 27 21:01:53.431206482 (3h23m16.660691839s) [upload=3507]: Upload finished. err: <nil> Duration:* 199.50337ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 442.033267ms FAIL (2025-08-27 21:27:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 27 21:01:36.226560503 (3h22m59.456045849s) [upload=3504]: Starting upload Aug 27 21:01:36.226622129 (3h22m59.456107476s) [upload=3504]: Done starting upload Aug 27 21:01:36.226666592 (3h22m59.456151949s) [upload=3504]: Uploading segments... Aug 27 21:01:36.226834699 (3h22m59.456320045s) [upload=3504]: Got next segment. Inline: false Aug 27 21:01:36.226865156 (3h22m59.456350502s) [upload=3504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:36.226869904 (3h22m59.456355252s) [upload=3504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:36.212957093 (3h23m59.44244245s) [upload=3504]: Next segment err: context deadline exceeded Aug 27 21:02:36.213033316 (3h23m59.442518673s) [upload=3504]: Done uploading segments... Aug 27 21:02:36.213048534 (3h23m59.442533891s) [upload=3504]: Upload finished. err: context deadline exceeded Duration: 442.056371ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 439.295516ms FAIL (2025-08-27 21:27:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 439.313508ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 189.140382ms FAIL (2025-08-27 21:27:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: accounting backend: Redis eval failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:07.426688152 (3h23m30.656173499s) [upload=3515]: Starting upload Aug 27 21:02:07.426735873 (3h23m30.65622122s) [upload=3515]: Done starting upload Aug 27 21:02:07.426790635 (3h23m30.656275992s) [upload=3515]: Uploading segments... Aug 27 21:02:07.426837303 (3h23m30.65632265s) [upload=3515]: Got next segment. Inline: true Aug 27 21:02:07.426841050 (3h23m30.656326387s) [upload=3515]: Waiting for error group managing segments... Aug 27 21:02:07.426872439 (3h23m30.656357786s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:07.426875605 (3h23m30.656360952s) [upload=3515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:02:07.426877649 (3h23m30.656362995s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:20.276735982 (3h23m43.506221359s) [upload=3515]: Done uploading segments... Aug 27 21:02:20.276756050 (3h23m43.506241427s) [upload=3515]: Upload finished. err: <nil> Duration:* 189.165688ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 195.718528ms FAIL (2025-08-27 21:27:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 195.739497ms core uplink us1
checkers.HTTP{"GET" "https://www.tardigrade.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.tardigrade_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 49.908019ms remote: "tcp:34.120.119.150:443" response headers: Referrer-Policy=no-referrer Content-Length=332 Date=Wed, 27 Aug 2025 21:14:56 GMT Alt-Svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Content-Type=text/html; charset=UTF-8 target: https://www.tardigrade.io/ FAIL (2025-08-27 21:27:18 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>502 Server Error</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Server Error</h1>\n<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n<h2></h2>\n</body></html>\n" Duration: 49.943396ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 534.830584ms FAIL (2025-08-27 21:27:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 534.850501ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 245.28383ms FAIL (2025-08-27 21:26:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 27 21:01:41.377970271 (3h23m4.607455648s) [upload=3507]: Starting upload Aug 27 21:01:41.378099755 (3h23m4.607585132s) [upload=3507]: Done starting upload Aug 27 21:01:41.378244187 (3h23m4.607729564s) [upload=3507]: Uploading segments... Aug 27 21:01:41.378310020 (3h23m4.607795397s) [upload=3507]: Got next segment. Inline: true Aug 27 21:01:41.378315851 (3h23m4.607801238s) [upload=3507]: Waiting for error group managing segments... Aug 27 21:01:41.378361376 (3h23m4.607846764s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:41.378366396 (3h23m4.607851773s) [upload=3507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:01:41.378370033 (3h23m4.607855409s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:01:53.431192035 (3h23m16.660677392s) [upload=3507]: Done uploading segments... Aug 27 21:01:53.431206482 (3h23m16.660691839s) [upload=3507]: Upload finished. err: <nil> Duration:* 245.296694ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 445.047957ms FAIL (2025-08-27 21:26:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 445.077562ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 449.107286ms FAIL (2025-08-27 21:26:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 27 21:01:36.226560503 (3h22m59.456045849s) [upload=3504]: Starting upload Aug 27 21:01:36.226622129 (3h22m59.456107476s) [upload=3504]: Done starting upload Aug 27 21:01:36.226666592 (3h22m59.456151949s) [upload=3504]: Uploading segments... Aug 27 21:01:36.226834699 (3h22m59.456320045s) [upload=3504]: Got next segment. Inline: false Aug 27 21:01:36.226865156 (3h22m59.456350502s) [upload=3504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:36.226869904 (3h22m59.456355252s) [upload=3504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:36.212957093 (3h23m59.44244245s) [upload=3504]: Next segment err: context deadline exceeded Aug 27 21:02:36.213033316 (3h23m59.442518673s) [upload=3504]: Done uploading segments... Aug 27 21:02:36.213048534 (3h23m59.442533891s) [upload=3504]: Upload finished. err: context deadline exceeded Duration: 449.127834ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.............X.....................................] duration: 423.934186ms FAIL (2025-08-27 21:26:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 423.953601ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 160.94866ms FAIL (2025-08-27 21:26:42 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: accounting backend: Redis eval failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:07.426688152 (3h23m30.656173499s) [upload=3515]: Starting upload Aug 27 21:02:07.426735873 (3h23m30.65622122s) [upload=3515]: Done starting upload Aug 27 21:02:07.426790635 (3h23m30.656275992s) [upload=3515]: Uploading segments... Aug 27 21:02:07.426837303 (3h23m30.65632265s) [upload=3515]: Got next segment. Inline: true Aug 27 21:02:07.426841050 (3h23m30.656326387s) [upload=3515]: Waiting for error group managing segments... Aug 27 21:02:07.426872439 (3h23m30.656357786s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:07.426875605 (3h23m30.656360952s) [upload=3515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:02:07.426877649 (3h23m30.656362995s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:20.276735982 (3h23m43.506221359s) [upload=3515]: Done uploading segments... Aug 27 21:02:20.276756050 (3h23m43.506241427s) [upload=3515]: Upload finished. err: <nil> Duration:* 160.966754ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://storj.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 42.157414ms remote: "tcp:34.120.119.150:443" response headers: Alt-Svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Content-Type=text/html; charset=UTF-8 Referrer-Policy=no-referrer Content-Length=332 Date=Wed, 27 Aug 2025 21:17:16 GMT target: https://storj.io/ FAIL (2025-08-27 21:26:38 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>502 Server Error</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Server Error</h1>\n<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n<h2></h2>\n</body></html>\n" Duration: 42.176239ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.665437283s FAIL (2025-08-27 21:26:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.665497967s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 202.962816ms FAIL (2025-08-27 21:26:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: accounting backend: Redis eval failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:07.426688152 (3h23m30.656173499s) [upload=3515]: Starting upload Aug 27 21:02:07.426735873 (3h23m30.65622122s) [upload=3515]: Done starting upload Aug 27 21:02:07.426790635 (3h23m30.656275992s) [upload=3515]: Uploading segments... Aug 27 21:02:07.426837303 (3h23m30.65632265s) [upload=3515]: Got next segment. Inline: true Aug 27 21:02:07.426841050 (3h23m30.656326387s) [upload=3515]: Waiting for error group managing segments... Aug 27 21:02:07.426872439 (3h23m30.656357786s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:07.426875605 (3h23m30.656360952s) [upload=3515]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:02:07.426877649 (3h23m30.656362995s) [upload=3515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:20.276735982 (3h23m43.506221359s) [upload=3515]: Done uploading segments... Aug 27 21:02:20.276756050 (3h23m43.506241427s) [upload=3515]: Upload finished. err: <nil> Duration:* 202.991151ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.............X....................................] duration: 778.058481ms FAIL (2025-08-27 21:26:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 8a256c146cd8f37955e142ac84d0faa38bd579ebb43ee626e10b5a173232e949 != before 4180454ac5ddc2c2ed56c97427218ce6f6f5d85b0003f9cbb032a89f4ea995f0 ======LOGS====== Aug 27 21:01:32.750878783 (3h22m55.98036416s) [upload=3502]: Starting upload Aug 27 21:01:32.750950248 (3h22m55.980435625s) [upload=3502]: Done starting upload Aug 27 21:01:32.751067818 (3h22m55.980553185s) [upload=3502]: Uploading segments... Aug 27 21:01:32.751334860 (3h22m55.980820237s) [upload=3502]: Got next segment. Inline: false Aug 27 21:01:32.751395275 (3h22m55.980880651s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:32.751400624 (3h22m55.980886012s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:21.290948080 (3h23m44.520433427s) [seg_pos={0 0}, upload=3502]: Begin upload segment... Aug 27 21:02:21.290984167 (3h23m44.520469524s) [seg_pos={0 0}, upload=3502]: Done begin upload segment. Aug 27 21:02:21.291006769 (3h23m44.520492127s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291142315 (3h23m44.520627692s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291344345 (3h23m44.520829702s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291696138 (3h23m44.521181515s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.291900602 (3h23m44.52138598s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292130163 (3h23m44.521615541s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292559302 (3h23m44.522044679s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292575773 (3h23m44.522061149s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.292633481 (3h23m44.522118858s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293075643 (3h23m44.52256101s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293144252 (3h23m44.522629628s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293448895 (3h23m44.522934272s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293757236 (3h23m44.523242622s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.293842716 (3h23m44.523328093s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294236097 (3h23m44.523721473s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294313281 (3h23m44.523798668s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.294949088 (3h23m44.524434464s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.295707735 (3h23m44.525193113s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Uploading piece... Aug 27 21:02:21.424581752 (3h23m44.654067109s) [upload=3502]: Next returned nil segment Aug 27 21:02:21.424595878 (3h23m44.654081245s) [upload=3502]: Waiting for error group managing segments... Aug 27 21:02:21.433819111 (3h23m44.663304479s) [noise=true , address=79.127.219.47:4056 , piece_id=RRCIXM76CJXQ57WYWAUXA6ACD64DJMCEKHKTTHPETPDLCGCWUIRA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.435597317 (3h23m44.665082684s) [noise=true , address=79.127.219.51:4197 , piece_id=56VI4UYTMCHF72PWZNFD2HTSX335N27CGVRKG7WC5JHD4BDGHG4Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.440452071 (3h23m44.669937439s) [noise=true , address=104.36.82.9:4038 , piece_id=CSPYJUZ5VGFK5DSIKIY5Q6DRLJJE4RJI4ZQ7OCKGELIUME7W3QVA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.443653585 (3h23m44.673138942s) [noise=true , address=104.36.82.41:4788 , piece_id=S7TS2ZIIVJG46WDR6JOB4DENJ7NW4WCZXW2UCXTNZKBTRHXRT2OQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.446290125 (3h23m44.675775473s) [noise=true , address=45.91.116.11:4124 , piece_id=Y4K6VNMU5DIEEUB2CIUJNZY63HLRYS2JZJXUTMTVA4NLNACCHWDQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.453133401 (3h23m44.682618768s) [noise=true , address=104.36.82.52:4088 , piece_id=4WJLZXA5ZC2XOKWC5XLIGWLBS66UPEXULKFMFY3RE47U6FVGQUGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.456748804 (3h23m44.686234181s) [noise=true , address=77.90.31.21:4897 , piece_id=EMB3LDMFCWRW4BKPNARAWGUYU5LFZ3KTLR7HS6F66JUL7T4FMLGQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.458665269 (3h23m44.688150636s) [noise=true , address=77.90.31.16:4738 , piece_id=V4GJFJMD6H55IJX4F3M7R27SCJCJ2EIV5JU6L6EKUKENQPT77DMA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.462817522 (3h23m44.692302889s) [noise=true , address=77.90.31.15:4708 , piece_id=D3PZXJEURKU57E5K5MECL3DOM2B42M5QUOVFJFD36WRT6UYCXT3Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.465059280 (3h23m44.694544657s) [noise=true , address=104.36.82.61:4378 , piece_id=YSROKHT2OELO3P57EIKKU4HVFL7PUMLZYZPQYRF75M3Q66OUL5DQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.467399463 (3h23m44.69688484s) [noise=true , address=104.36.82.16:4315 , piece_id=TAVXG3NTXLYYR4ZAZE3O476ZKGGXVMSJ2J32UGMCJ5XCR2RV5WQA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.471858793 (3h23m44.701344171s) [noise=true , address=104.36.82.37:4687 , piece_id=H2CXH33EV4SV7RB7B6WOOLH72KZKIE36PD4RL5F6XBCXTQ26NGVQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.480471758 (3h23m44.709957145s) [noise=true , address=104.36.82.11:4125 , piece_id=T2L6SJ3AIZYWDLJHMCKFQNFVE2MX2RRCF7WOGQBLFFDAV4FTIRUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568716922 (3h23m44.798202269s) [noise=true , address=77.90.31.19:4853 , piece_id=2UQSONX56FRWWV3JCIMOB7CU3WFUX54EHGZ5442RFF3UU3SKQGYA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: <nil> Aug 27 21:02:21.568741268 (3h23m44.798226625s) [seg_pos={0 0}, upload=3502]: Segment reached optimal threshold of 14 pieces. Aug 27 21:02:21.568847017 (3h23m44.798332374s) [noise=true , address=79.127.201.215:4358 , piece_id=AUKEJZCQO4S5UOSQ3QLLCBSWNZLF7B4SOINJW4OMY5MPHIYLVW2Q, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cy2na9tb5e6cEXSiM9E4yuZszRDNv4gQU7VBUXnQCSRoPHTA6): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568907941 (3h23m44.798393288s) [noise=true , address=77.90.31.10:4521 , piece_id=SVHWKNNKEVII3E3ABZGFDR2EOS7WRS6ATG3LKARMGNLGQZXWBTNQ, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568944320 (3h23m44.798429677s) [noise=true , address=79.127.205.245:4221 , piece_id=BGIYKF74JI7SDRQRJSPFVLW2HSTS7THZCO5372JY7XYY2TC4VXUA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mRTHdVPMGp5pANgUX5B657RvXmBo9fD1JfLJHNsYGRnPgczZ4): protocol: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:48984->79.127.205.245:4221: use of closed network connection Aug 27 21:02:21.568961932 (3h23m44.798447289s) [noise=true , address=77.90.31.17:4781 , piece_id=RM72FQ2B5N4AMUSHD5BL7CDSIUQ2FVRYMIXIQH3UGYSJMW2XZXDA, seg_pos={0 0}, upload=3502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:21.568983242 (3h23m44.7984686s) [seg_pos={0 0}, upload=3502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:02:21.569081298 (3h23m44.798566655s) [upload=3502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:21.569085866 (3h23m44.798571223s) [upload=3502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:29.941659943 (3h23m53.1711453s) [upload=3502]: Done uploading segments... Aug 27 21:02:29.941674550 (3h23m53.171159907s) [upload=3502]: Upload finished. err: <nil> Duration: 778.081074ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://www.tardigrade.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.tardigrade_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 41.601712ms remote: "tcp:34.120.119.150:443" response headers: Referrer-Policy=no-referrer Content-Length=332 Date=Wed, 27 Aug 2025 21:14:56 GMT Alt-Svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Content-Type=text/html; charset=UTF-8 target: https://www.tardigrade.io/ FAIL (2025-08-27 21:26:11 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>502 Server Error</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Server Error</h1>\n<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n<h2></h2>\n</body></html>\n" Duration: 41.633461ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 471.341617ms FAIL (2025-08-27 21:26:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:02:18.742567354 (3h23m41.972052711s) [upload=3517]: Starting upload Aug 27 21:02:18.742644770 (3h23m41.972130117s) [upload=3517]: Done starting upload Aug 27 21:02:18.742774504 (3h23m41.972259851s) [upload=3517]: Uploading segments... Aug 27 21:02:18.742916050 (3h23m41.972401407s) [upload=3517]: Got next segment. Inline: false Aug 27 21:02:18.742942710 (3h23m41.972428057s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:02:18.742946277 (3h23m41.972431634s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:22.763776455 (3h23m45.993261832s) [seg_pos={0 0}, upload=3517]: Begin upload segment... Aug 27 21:02:22.763826518 (3h23m45.993311906s) [seg_pos={0 0}, upload=3517]: Done begin upload segment. Aug 27 21:02:22.763856284 (3h23m45.993341662s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.763877575 (3h23m45.993362953s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764397404 (3h23m45.99388278s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764422130 (3h23m45.993907507s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764862849 (3h23m45.994348237s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.764927922 (3h23m45.994413298s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765295312 (3h23m45.99478069s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765414958 (3h23m45.994900335s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.765749918 (3h23m45.995235295s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766207238 (3h23m45.995692616s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766610447 (3h23m45.996095824s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.766966717 (3h23m45.996452094s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767231696 (3h23m45.996717073s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767550926 (3h23m45.997036283s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.767825002 (3h23m45.99731036s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.768089159 (3h23m45.997574507s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Uploading piece... Aug 27 21:02:22.887072978 (3h23m46.116558345s) [upload=3517]: Next returned nil segment Aug 27 21:02:22.887088627 (3h23m46.116573994s) [upload=3517]: Waiting for error group managing segments... Aug 27 21:02:22.896000215 (3h23m46.125485581s) [noise=true , address=45.91.116.6:4974 , piece_id=5QXPGI3XYTQL32VUXYMOQ54XLUJEAHZBIBONW767YOT43RA6PBIQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.897639208 (3h23m46.127124576s) [noise=true , address=45.91.116.12:4227 , piece_id=SQKEVOYWMFBKXCMGFBT7DIYTN7QS4UBRGCFNRLCNWPGHFOJA7XJQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.898268773 (3h23m46.12775413s) [noise=true , address=45.91.116.12:4225 , piece_id=FZGNYKDMAQAYD5XAF2N6SIJMQLA46PVXPN6T35JLSAUQUUCBAYCA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.899821925 (3h23m46.129307282s) [noise=true , address=45.91.116.9:4261 , piece_id=IK5OIGTBU723YGGDN3RPXKUVZ3QC3NEM3T27DYT5LDHW3B4SYPKQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900364926 (3h23m46.129850283s) [noise=true , address=45.91.116.4:4938 , piece_id=SQRBXTCCYOWGL2VAFM75RPTBQFPW6IFY4R5EHYZRF5G6FUPVNMNA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.900542851 (3h23m46.130028228s) [noise=true , address=45.91.116.10:4083 , piece_id=KBS4UACDAU2GLY4JTEDO7SDI2U66Z3EYDIUAZ2IYI5ET5SUTX6CQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.901166505 (3h23m46.130651862s) [noise=true , address=77.90.31.10:4518 , piece_id=46XH35VERNQPW4RJ5ZOHHPH2MU3ADGS6FNOPLOZLPV66MTJDYP5Q, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.902796171 (3h23m46.132281538s) [noise=true , address=104.36.82.9:4005 , piece_id=B5TH2PWMLINVCJI4HVOYWS6B66KIUNPHHY52HWF5O5IB34QFFFPA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.904269574 (3h23m46.13375491s) [noise=true , address=45.91.116.7:4009 , piece_id=UKIZPHON3RLITQ2CWMFWDF7M26EG5DCDDFMV5EQBGK2G6J5ONNMQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.905707979 (3h23m46.135193326s) [noise=true , address=104.36.82.2:4731 , piece_id=MJCWMSTTWQHZOHA5P42TFJDK3QRRFUQ6KN4A6ZBDHMA53AWBRPSA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.907552551 (3h23m46.137037897s) [noise=true , address=104.36.82.15:4259 , piece_id=IDJKXXMXEGK33JGUNUU6XQTDKW2YUL2XTHIVQRLAVMDA7Q6LE75A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.908524448 (3h23m46.138009795s) [noise=true , address=77.90.31.13:4627 , piece_id=LOX4EMR3AU76ALQ7P6RGS6O7ROXVFNQHTPTQTEU2OHU7DKVKMTEA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914085382 (3h23m46.143570729s) [noise=true , address=104.36.82.6:4873 , piece_id=S43FRJTNLACGGL4IREGGZDRDOAL2MBQGCOBYH66WMYQ6QQ6DDPEQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: <nil> Aug 27 21:02:22.914102324 (3h23m46.143587671s) [seg_pos={0 0}, upload=3517]: Segment reached optimal threshold of 13 pieces. Aug 27 21:02:22.914233530 (3h23m46.143718877s) [noise=true , address=104.36.82.16:4294 , piece_id=JJZDI2PNUG5YZ6CSNHGTLYCRN5LW7BWWVH2FRHVH67FWSFVMGDGA, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914300046 (3h23m46.143785392s) [noise=true , address=77.90.31.11:4555 , piece_id=FIRU7P4OW3CSYXWTTTMHYNZFN22R52FJH7I67PVB4FI65FND72VQ, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914345892 (3h23m46.143831249s) [noise=true , address=45.91.116.8:4045 , piece_id=DRKMDKC2XKXARK3XAYY6LP62BRPRB7NDL2B63QS3WKCLK732IX6A, seg_pos={0 0}, upload=3517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 27 21:02:22.914401045 (3h23m46.143886412s) [seg_pos={0 0}, upload=3517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 27 21:02:22.914505472 (3h23m46.143990829s) [upload=3517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:02:22.914511263 (3h23m46.143996629s) [upload=3517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:24.939283761 (3h23m48.168769138s) [upload=3517]: Done uploading segments... Aug 27 21:02:24.939309329 (3h23m48.168794706s) [upload=3517]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 471.369128ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.787415153s FAIL (2025-08-27 21:26:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.78744061s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 552.299382ms FAIL (2025-08-27 21:26:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 27 21:01:36.226560503 (3h22m59.456045849s) [upload=3504]: Starting upload Aug 27 21:01:36.226622129 (3h22m59.456107476s) [upload=3504]: Done starting upload Aug 27 21:01:36.226666592 (3h22m59.456151949s) [upload=3504]: Uploading segments... Aug 27 21:01:36.226834699 (3h22m59.456320045s) [upload=3504]: Got next segment. Inline: false Aug 27 21:01:36.226865156 (3h22m59.456350502s) [upload=3504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:36.226869904 (3h22m59.456355252s) [upload=3504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:02:36.212957093 (3h23m59.44244245s) [upload=3504]: Next segment err: context deadline exceeded Aug 27 21:02:36.213033316 (3h23m59.442518673s) [upload=3504]: Done uploading segments... Aug 27 21:02:36.213048534 (3h23m59.442533891s) [upload=3504]: Upload finished. err: context deadline exceeded Duration: 552.330432ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 204.53846ms FAIL (2025-08-27 21:26:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 27 21:01:41.377970271 (3h23m4.607455648s) [upload=3507]: Starting upload Aug 27 21:01:41.378099755 (3h23m4.607585132s) [upload=3507]: Done starting upload Aug 27 21:01:41.378244187 (3h23m4.607729564s) [upload=3507]: Uploading segments... Aug 27 21:01:41.378310020 (3h23m4.607795397s) [upload=3507]: Got next segment. Inline: true Aug 27 21:01:41.378315851 (3h23m4.607801238s) [upload=3507]: Waiting for error group managing segments... Aug 27 21:01:41.378361376 (3h23m4.607846764s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:41.378366396 (3h23m4.607851773s) [upload=3507]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 21:01:41.378370033 (3h23m4.607855409s) [upload=3507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:01:53.431192035 (3h23m16.660677392s) [upload=3507]: Done uploading segments... Aug 27 21:01:53.431206482 (3h23m16.660691839s) [upload=3507]: Upload finished. err: <nil> Duration:* 204.569389ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 187.915716ms FAIL (2025-08-27 21:25:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 187.964088ms core uplink us1
checkers.HTTP{"GET" "https://www.tardigrade.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.tardigrade_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 45.514702ms remote: "tcp:34.120.119.150:443" response headers: Referrer-Policy=no-referrer Content-Length=332 Date=Wed, 27 Aug 2025 21:14:56 GMT Alt-Svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Content-Type=text/html; charset=UTF-8 target: https://www.tardigrade.io/ FAIL (2025-08-27 21:25:44 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>502 Server Error</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Server Error</h1>\n<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n<h2></h2>\n</body></html>\n" Duration: 45.540009ms core website
checkers.HTTP{"GET" "https://storj.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 52.008201ms remote: "tcp:34.120.119.150:443" response headers: Alt-Svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Content-Type=text/html; charset=UTF-8 Referrer-Policy=no-referrer Content-Length=332 Date=Wed, 27 Aug 2025 21:17:16 GMT target: https://storj.io/ FAIL (2025-08-27 21:25:44 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>502 Server Error</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Server Error</h1>\n<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n<h2></h2>\n</body></html>\n" Duration: 52.032065ms core website

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://www.tardigrade.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.tardigrade_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://storj.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://www.tardigrade.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.tardigrade_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://storj.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://www.tardigrade.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.tardigrade_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://www.tardigrade.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.tardigrade_io)

checkers.HTTP{"GET" "https://storj.io/", ExpectedStatus: 301, ExpectedRedirect: "https://www.storj.io/"} failure (antiblip window 6/15): (http.storj_io)