Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
16 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX............................................] duration:
776.467761ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:44:35 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 776.490714ms
core
satellite
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
14 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX..............................................] duration:
283.87676ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:44:32 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 283.924579ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
11 out of 60 is failed (threshold=6) [X.XXXXXXXX.XX...............................................] duration:
266.888842ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:44:16 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 266.906396ms
edge
linksharing
bgp
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
14 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXX.............................................] duration:
288.746609ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:44:13 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 288.764112ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
2.22185185s
FAIL (2025-11-21 12:44:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.221876686s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
10 out of 60 is failed (threshold=3) [XXXXXXXXXX..................................................] duration:
2.105982648s
FAIL (2025-11-21 12:43:55 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.106001554s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX...........................................] duration:
736.619229ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:43:44 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 736.64109ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
2.118113661s
FAIL (2025-11-21 12:43:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.118138287s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX..........................................] duration:
4.655575747s
FAIL (2025-11-21 12:43:31 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.655595233s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
11 out of 60 is failed (threshold=3) [XXXXXXXXXXX.................................................] duration:
2.234109922s
FAIL (2025-11-21 12:43:21 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.234133286s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
15 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXX............................................] duration:
286.814254ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:43:08 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 286.839842ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
12 out of 60 is failed (threshold=6) [XX.XXXXXXXX.XX..............................................] duration:
272.791891ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:43:05 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 272.819763ms
edge
linksharing
bgp
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX..........................................] duration:
798.852747ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:43:00 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 798.882824ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
2.102668722s
FAIL (2025-11-21 12:42:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.102693146s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
12 out of 60 is failed (threshold=3) [XXXXXXXXXXXX................................................] duration:
2.174983434s
FAIL (2025-11-21 12:42:52 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.175011358s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.........................................] duration:
4.857408794s
FAIL (2025-11-21 12:42:49 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.857451003s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
13 out of 60 is failed (threshold=6) [XXX.XXXXXXXX.XX.............................................] duration:
828.375372ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:42:47 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 828.398897ms
edge
linksharing
bgp
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
13 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX...............................................] duration:
2.124129008s
FAIL (2025-11-21 12:42:24 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.124151811s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
16 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXXXXX...........................................] duration:
280.492028ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:42:21 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 280.511085ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.........................................] duration:
793.333236ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:42:19 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 793.378963ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
20 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX........................................] duration:
4.62004734s
FAIL (2025-11-21 12:42:13 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.62006845s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
17 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXXXXXX..........................................] duration:
280.172474ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:42:04 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 280.197211ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
13 out of 60 is failed (threshold=6) [.XXX.XXXXXXXX.XX............................................] duration:
284.691016ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:42:00 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 284.711585ms
edge
linksharing
bgp
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
2.165699452s
FAIL (2025-11-21 12:41:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.165722486s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
14 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX..............................................] duration:
2.104555074s
FAIL (2025-11-21 12:41:41 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.104580614s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.......................................] duration:
4.389918345s
FAIL (2025-11-21 12:41:35 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.38994185s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
14 out of 60 is failed (threshold=6) [X.XXX.XXXXXXXX.XX...........................................] duration:
271.860617ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:41:28 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 271.886385ms
edge
linksharing
bgp
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
15 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.............................................] duration:
2.200082743s
FAIL (2025-11-21 12:41:22 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.200103573s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
2.279226264s
FAIL (2025-11-21 12:41:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.279249388s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
20 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX........................................] duration:
811.806529ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:41:05 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 811.829752ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
16 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX............................................] duration:
2.092736443s
FAIL (2025-11-21 12:41:04 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.09276147s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX......................................] duration:
4.846543916s
FAIL (2025-11-21 12:40:58 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.846569344s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
17 out of 60 is failed (threshold=3) [.XXX.XXXXXXXXXXXXXX.........................................] duration:
286.897644ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:40:50 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 286.915888ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
2.135923696s
FAIL (2025-11-21 12:40:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.13594232s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
18 out of 60 is failed (threshold=3) [X.XXX.XXXXXXXXXXXXXX........................................] duration:
269.527196ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:40:34 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 269.558676ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
15 out of 60 is failed (threshold=6) [XX.XXX.XXXXXXXX.XX..........................................] duration:
274.407207ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:40:31 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 274.425371ms
edge
linksharing
bgp
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
19 out of 60 is failed (threshold=3) [XX.XXX.XXXXXXXXXXXXXX.......................................] duration:
292.318897ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:40:16 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 292.339896ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
2.163482721s
FAIL (2025-11-21 12:40:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.163508409s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX...........................................] duration:
2.248835394s
FAIL (2025-11-21 12:40:10 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.248859429s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.....................................] duration:
4.840915122s
FAIL (2025-11-21 12:39:55 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.840933135s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.......................................] duration:
810.784298ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:39:53 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 810.798163ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.151228492s
FAIL (2025-11-21 12:39:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
..
Nov 21 12:26:59.829635995 (601h13m26.53058088s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829762222 (601h13m26.530707117s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.829922452 (601h13m26.530867337s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830067675 (601h13m26.53101257s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830221394 (601h13m26.531166289s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830406341 (601h13m26.531351225s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.830719097 (601h13m26.531663992s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831043777 (601h13m26.531988672s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831363807 (601h13m26.532308692s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.831725285 (601h13m26.53267018s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:26:59.832112532 (601h13m26.533057427s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Uploading piece...
Nov 21 12:27:00.275101242 (601h13m26.976046127s) [upload=528721]: Next returned nil segment
Nov 21 12:27:00.275113505 (601h13m26.97605839s) [upload=528721]: Waiting for error group managing segments...
Nov 21 12:27:00.473175900 (601h13m27.174120795s) [noise=true , address=81.171.14.70:4033 , piece_id=MRIUDQA33E57BE7BTUFY3VM6LGNPKI27BIC655N4CFSFO5XIV3QA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.492220637 (601h13m27.193165532s) [noise=true , address=89.149.201.147:4005 , piece_id=YE52542D55PRH4R7WRHM7H5SESV6ADUIDHTM7Z5Q4OWQY2XUOYHA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.494560869 (601h13m27.195505754s) [noise=true , address=185.167.204.205:4007 , piece_id=T5XFQHJFUMVPC747BKENK2JANECD5M7KHZETHGYZLJEME7G63XNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.497798975 (601h13m27.19874386s) [noise=true , address=185.167.204.205:4033 , piece_id=S56UO6477KPFEZ7N4A72IJYB5NRDUQGH2PITHU6CRK6IR6XJ65JA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.500087008 (601h13m27.201031903s) [noise=true , address=185.167.204.206:4070 , piece_id=TVBVWVSJSELXQRYWYTXFCIHY4DJNS2DWZSN4SLQFDLXTO4WCGPNQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.504753255 (601h13m27.20569815s) [noise=true , address=89.149.201.147:4001 , piece_id=7PNK2S5MNODTGZV3EVDWENHC3HERQMJQXHGBB743XBWSCG344P5Q, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.505622837 (601h13m27.206567732s) [noise=true , address=185.167.204.206:4062 , piece_id=FMNU2EQ3UIH7S2LE2PJG37UFCMADMNKO6FOIWKYLXB3HRBBTPOPA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.511127325 (601h13m27.212072211s) [noise=true , address=89.149.201.147:4004 , piece_id=UGCIH3OD7ZRPCKAF37GFQURKQ3OBUBMHDCA6V45GQQIKHK5TYGQA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.516154990 (601h13m27.217099874s) [noise=true , address=81.171.14.70:4041 , piece_id=X7YUS2GQIUI75SSFTJWPY3JMI53Z6NDKAB4QY36YYSGJWYPZ3RYA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.527479841 (601h13m27.228424736s) [noise=true , address=185.167.204.206:4053 , piece_id=PSGPVNMBGPJKK5DVOZRC63SXGLKVELTOYYN5M7E3XRA742UX3X5A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.529960096 (601h13m27.23090498s) [noise=true , address=81.171.14.70:4039 , piece_id=JWYZDKYJIL72EHTIIT2GCO3KCYHSO7VJU5BOHTF5ZZ3ESSRQB3TQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.531717714 (601h13m27.232662609s) [noise=true , address=185.167.204.205:4024 , piece_id=UUMPXYMI4BH5W2JNQRPECF6HKFZ6V6W7RAAEX6DJHZ7INSTTUU6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533125465 (601h13m27.23407036s) [noise=true , address=81.171.14.70:4036 , piece_id=2TEENBTP5A4GS22IFSIDEHEAJROOCJAWNE22N5H67EVFB44XPDNA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.533634170 (601h13m27.234579065s) [noise=true , address=89.149.201.147:4016 , piece_id=WHD4JJOY6GH45Y6NXSJRV4SJYTNTYA5RZBHLDYN3I7SNVRWHFYLA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535694085 (601h13m27.23663898s) [noise=true , address=81.171.14.70:4048 , piece_id=HDFF2UIPIQZZRK3MBCLBOL35KAMSDF6PWSWJGGOVI7BJTEPRTBPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.535874314 (601h13m27.236819198s) [noise=true , address=185.167.204.205:4006 , piece_id=2KRM3PDYYWHWSMO5JHZQHWU2WDFFRIYYTOZEPZKAGSWUANLMUBRA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.536978726 (601h13m27.237923621s) [noise=true , address=185.167.204.206:4038 , piece_id=IGHESWRCMGLHAK5LJMDCZYTB3VK4IXNPCHAO7E5PBCMQRNSF6E6A, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540094382 (601h13m27.241039267s) [noise=true , address=81.171.14.70:4042 , piece_id=57GJHADFJT47YM672HWDEK3LXGQJEM6LGMWC7GOLIJYL4HTTXNTA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.540264512 (601h13m27.241209407s) [noise=true , address=185.167.204.205:4002 , piece_id=J7XHA6GGDXJVTSAJVEBADZO24ZHBUTZIUUF5YBFTPE7Y4PMHRFLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.545151602 (601h13m27.246096498s) [noise=true , address=81.171.14.70:4030 , piece_id=TZ6UPMEB3D33NU5ZJ2N22IDAXLAX5FYVMID2CVMHAQZW5YET5OLQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.568219719 (601h13m27.269164614s) [noise=true , address=81.171.14.70:4025 , piece_id=AMUN7C4TYI6E7RHYCNFOOMUHJVXQWMMMGWHI4SX63HJ7MXAEUWVQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.580993630 (601h13m27.281938525s) [noise=true , address=89.149.201.147:4018 , piece_id=6IGAUQBR4S4MYQB6GUFGZ6H3O6BM6JRH6CZJWADI4IWIADCZCVJA, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.601442703 (601h13m27.302387598s) [noise=true , address=89.149.201.147:4022 , piece_id=RZKOW2DK63RXE5DUBXNCWGU3674DUV2IIX5UJ5HZCXNSZOLALTXQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:00.621696008 (601h13m27.322640893s) [noise=true , address=185.167.204.206:4045 , piece_id=IK455E5SYS7GYAZKBTUM5TQONQOHONULLPXLAXMQ3KA27MBU3IPQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: <nil>
Nov 21 12:27:59.419543571 (601h14m26.120488456s) [noise=true , address=185.167.204.205:4021 , piece_id=5P2W7XW6AJUQL6CDVXKYVFSEA4YIRSGUGBT2GD76RATLWGBILIAQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419597923 (601h14m26.120542818s) [noise=true , address=81.171.14.70:4035 , piece_id=75LACO6Y5NQHJQCSQELRI2FFZL5P4I6GOZO52F3T7DLJSRKPO6ZQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: rpc: dial tcp 81.171.14.70:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419636766 (601h14m26.120581661s) [noise=true , address=185.167.204.205:4026 , piece_id=RMJSTZEHHCILUKPPOZHH2Z2CG6MM3OKS7TSD2XLWDOUHS4JLGGRQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: rpc: dial tcp 185.167.204.205:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419683043 (601h14m26.120627928s) [noise=true , address=185.167.204.206:4037 , piece_id=PW4C3KRTOYQKGKBJYM7IQTKAVHILHVRY2EVVUQ4EK3ZLO5NIERDQ, seg_pos={0 0}, upload=528721]: Done uploading piece. err: ecclient: upload failed (node:12nEq1PMWr39K5LBUcViEtEb1KTnsJLjMaBbNGuqpc13WdZb18R, address:185.167.204.206:4037): protocol: write tcp 5.161.89.252:55498->185.167.204.206:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 21 12:27:59.419700365 (601h14m26.12064526s) [seg_pos={0 0}, upload=528721]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 21 12:27:59.527560845 (601h14m26.22850574s) [upload=528721]: Done uploading segments...
Nov 21 12:27:59.527579510 (601h14m26.228524405s) [upload=528721]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 2.151250154s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
16 out of 60 is failed (threshold=6) [XXX.XXX.XXXXXXXX.XX.........................................] duration:
282.46166ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:39:41 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 282.485745ms
edge
linksharing
bgp
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX......................................] duration:
804.00128ms remote:
"tcp:34.2.157.232:443" target:
https://ap1.storj.io/login
FAIL (2025-11-21 12:39:26 UTC)
orbiter warning: Get "https://ap1.storj.io/login": read tcp 5.161.89.252:12384->34.2.157.232:443: read: connection reset by peer
Duration: 804.023181ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX..........................................] duration:
2.274431308s
FAIL (2025-11-21 12:39:19 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.274445034s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
20 out of 60 is failed (threshold=3) [XXX.XXX.XXXXXXXXXXXXXX......................................] duration:
294.00368ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:39:17 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 294.023657ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX....................................] duration:
4.889945228s
FAIL (2025-11-21 12:39:11 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.889965064s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
17 out of 60 is failed (threshold=6) [XXXX.XXX.XXXXXXXX.XX........................................] duration:
306.545163ms remote:
"tcp:136.0.77.2:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt
FAIL (2025-11-21 12:39:08 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 306.589436ms
edge
linksharing
bgp
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.........................................] duration:
2.10420719s
FAIL (2025-11-21 12:38:51 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.104235342s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
21 out of 60 is failed (threshold=3) [XXXX.XXX.XXXXXXXXXXXXXX.....................................] duration:
304.185376ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-11-21 12:38:36 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": context deadline exceeded
Duration: 304.200645ms
edge
gateway
vhost
bgp
ap1
|