Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
8 out of 60 is failed (threshold=3) [XXXXXXXX....................................................] duration:
4.613283431s
FAIL (2025-11-21 12:51:04 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.613305493s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
2.122395088s
FAIL (2025-11-21 12:50:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.122416278s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
2.227477835s
FAIL (2025-11-21 12:50:52 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.227501769s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
4 out of 60 is failed (threshold=6) [XX.XX.......................................................] duration:
305.401241ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:50:49 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 305.418964ms
edge
linksharing
bgp
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
9 out of 60 is failed (threshold=3) [XXXXXXXXX...................................................] duration:
735.881904ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:50:24 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 735.908604ms
core
satellite
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
6 out of 60 is failed (threshold=3) [XXXXXX......................................................] duration:
275.274808ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:50:24 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 275.294405ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
2.201148471s
FAIL (2025-11-21 12:50:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.201171805s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
7 out of 60 is failed (threshold=3) [XXXXXXX.....................................................] duration:
270.379955ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:50:04 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 270.400344ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [XXX.........................................................] duration:
2.082601072s
FAIL (2025-11-21 12:49:59 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.082625027s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
9 out of 60 is failed (threshold=3) [XXXXXXXXX...................................................] duration:
4.726840544s
FAIL (2025-11-21 12:49:58 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.726859648s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
5 out of 60 is failed (threshold=6) [XXX.XX......................................................] duration:
282.22802ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:49:42 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 282.24959ms
edge
linksharing
bgp
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
10 out of 60 is failed (threshold=3) [XXXXXXXXXX..................................................] duration:
770.845049ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:49:10 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 770.864997ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
2.223832165s
FAIL (2025-11-21 12:49:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.22386126s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
4 out of 60 is failed (threshold=3) [XXXX........................................................] duration:
2.078169407s
FAIL (2025-11-21 12:48:58 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.078194885s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
10 out of 60 is failed (threshold=3) [XXXXXXXXXX..................................................] duration:
4.743941751s
FAIL (2025-11-21 12:48:57 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.743961879s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
8 out of 60 is failed (threshold=3) [XXXXXXXX....................................................] duration:
280.805126ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:48:57 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 280.867583ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
6 out of 60 is failed (threshold=6) [XXXX.XX.....................................................] duration:
264.237028ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:48:49 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 264.259121ms
edge
linksharing
bgp
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
11 out of 60 is failed (threshold=3) [XXXXXXXXXXX.................................................] duration:
738.167193ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:48:22 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 738.240401ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
2.166594031s
FAIL (2025-11-21 12:48:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.166620902s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
9 out of 60 is failed (threshold=3) [XXXXXXXXX...................................................] duration:
276.270945ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:48:15 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 276.28974ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
11 out of 60 is failed (threshold=3) [XXXXXXXXXXX.................................................] duration:
4.630354663s
FAIL (2025-11-21 12:48:04 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.630378268s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration:
2.35453384s
FAIL (2025-11-21 12:47:58 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.354564578s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
7 out of 60 is failed (threshold=6) [XXXXX.XX....................................................] duration:
275.904852ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:47:57 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 275.928697ms
edge
linksharing
bgp
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
2.215733192s
FAIL (2025-11-21 12:47:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.215753029s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
12 out of 60 is failed (threshold=3) [XXXXXXXXXXXX................................................] duration:
767.494531ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:47:32 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 767.512324ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
6 out of 60 is failed (threshold=3) [XXXXXX......................................................] duration:
2.123587853s
FAIL (2025-11-21 12:47:23 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.12361279s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
10 out of 60 is failed (threshold=3) [XXXXXXXXXX..................................................] duration:
295.23909ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:47:22 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 295.256754ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
12 out of 60 is failed (threshold=3) [XXXXXXXXXXXX................................................] duration:
4.705465535s
FAIL (2025-11-21 12:47:16 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.705489018s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
8 out of 60 is failed (threshold=6) [XXXXXX.XX...................................................] duration:
270.196807ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:47:10 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 270.215242ms
edge
linksharing
bgp
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
7 out of 60 is failed (threshold=3) [XXXXXXX.....................................................] duration:
2.191452677s
FAIL (2025-11-21 12:47:01 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.191473156s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
13 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX...............................................] duration:
711.93468ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:46:55 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 711.982949ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
13 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX...............................................] duration:
4.750433463s
FAIL (2025-11-21 12:46:42 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.750455504s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
2.159061741s
FAIL (2025-11-21 12:46:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.159084955s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
14 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX..............................................] duration:
756.817513ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:46:39 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 756.835887ms
core
satellite
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
9 out of 60 is failed (threshold=6) [XXXXXXX.XX..................................................] duration:
287.422642ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:46:37 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 287.444593ms
edge
linksharing
bgp
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
2.149486081s
FAIL (2025-11-21 12:46:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.149506488s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
11 out of 60 is failed (threshold=3) [XXXXXXXXXXX.................................................] duration:
786.108651ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:46:17 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 786.12366ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
14 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX..............................................] duration:
4.709964405s
FAIL (2025-11-21 12:46:14 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.709988661s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
15 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.............................................] duration:
4.769086393s
FAIL (2025-11-21 12:45:52 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.769104517s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
8 out of 60 is failed (threshold=3) [XXXXXXXX....................................................] duration:
2.152776061s
FAIL (2025-11-21 12:45:49 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.152825034s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
12 out of 60 is failed (threshold=3) [XXXXXXXXXXXX................................................] duration:
290.554632ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:45:46 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 290.567826ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
2.236425256s
FAIL (2025-11-21 12:45:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.236448068s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
10 out of 60 is failed (threshold=6) [XXXXXXXX.XX.................................................] duration:
276.084448ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:45:40 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 276.105469ms
edge
linksharing
bgp
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
15 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.............................................] duration:
779.853535ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:45:39 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 779.872761ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
16 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX............................................] duration:
4.839561819s
FAIL (2025-11-21 12:45:18 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.839579592s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
10 out of 60 is failed (threshold=6) [.XXXXXXXX.XX................................................] duration:
284.281952ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:45:06 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 284.298512ms
edge
linksharing
bgp
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
13 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX...............................................] duration:
279.9706ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:45:06 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 279.988794ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
2.246541087s
FAIL (2025-11-21 12:45:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.246565222s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
9 out of 60 is failed (threshold=3) [XXXXXXXXX...................................................] duration:
2.166939155s
FAIL (2025-11-21 12:44:52 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.166961547s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX...........................................] duration:
4.589953831s
FAIL (2025-11-21 12:44:42 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.589988476s
core
uplink
ap1
|