Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.............................................XX.............] duration:
5.019029541s
FAIL (2025-09-18 16:57:31 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 5.019054649s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [.............................................XX.............] duration:
754.071823ms target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:57:27 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 754.097051ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..............................................XX............] duration:
5.047116794s
FAIL (2025-09-18 16:57:11 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 5.047144716s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [...............................................XX...........] duration:
5.14579085s
FAIL (2025-09-18 16:56:51 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 5.145818001s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [..............................................XXX...........] duration:
2.158804845s
FAIL (2025-09-18 16:56:48 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.158841113s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [..............................................XX............] duration:
737.5608ms target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:56:21 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 737.589834ms
core
satellite
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [...............................................XX...........] duration:
754.001914ms target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:56:00 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 754.037181ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [................................................XX..........] duration:
4.965597931s
FAIL (2025-09-18 16:55:46 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.965626043s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [...............................................XXX..........] duration:
2.196388428s
FAIL (2025-09-18 16:55:43 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.196422803s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [................................................XX..........] duration:
744.139083ms target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:55:38 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 744.160233ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [................................................XXX.........] duration:
2.095945966s
FAIL (2025-09-18 16:55:10 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.095978728s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.................................................XX.........] duration:
5.257560868s
FAIL (2025-09-18 16:55:01 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 5.257579123s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [.................................................XXX........] duration:
2.312779052s
FAIL (2025-09-18 16:54:54 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.312798499s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [.................................................XX.........] duration:
729.669561ms target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:54:44 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 729.690901ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..................................................XX........] duration:
5.335720213s
FAIL (2025-09-18 16:54:43 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 5.335744848s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [..................................................XXX.......] duration:
2.152430823s
FAIL (2025-09-18 16:54:24 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.152468494s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [...................................................XX.......] duration:
5.812871274s
FAIL (2025-09-18 16:53:45 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 5.81289581s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [..................................................XX........] duration:
740.669901ms target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:53:40 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 740.686382ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [....................................................XX......] duration:
5.141775164s
FAIL (2025-09-18 16:53:29 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 5.141802695s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [...................................................XX.......] duration:
732.948861ms target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:53:04 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 732.966084ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration:
5.035480836s
FAIL (2025-09-18 16:52:37 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 5.035505041s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [....................................................XX......] duration:
752.597297ms target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:52:29 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 752.611803ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [....................................................XXX.....] duration:
2.265761686s
FAIL (2025-09-18 16:52:25 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.265785862s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration:
783.074606ms target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:52:03 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 783.095757ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [......................................................XX....] duration:
6.085689109s
FAIL (2025-09-18 16:51:26 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 6.085711471s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [.....................................................XXX....] duration:
2.22218947s
FAIL (2025-09-18 16:51:20 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.222211752s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [......................................................XX....] duration:
756.990832ms target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:50:52 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 757.007002ms
core
satellite
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................XX...] duration:
788.184694ms target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:50:22 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 788.20338ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................XX...] duration:
5.60383179s
FAIL (2025-09-18 16:50:21 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 5.603860375s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [......................................................XXX...] duration:
2.250725446s
FAIL (2025-09-18 16:50:17 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.25074299s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [........................................................XX..] duration:
5.044021167s
FAIL (2025-09-18 16:49:40 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 5.0440489s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [........................................................XX..] duration:
819.601184ms target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:49:37 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 819.627873ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration:
2.17084834s
FAIL (2025-09-18 16:49:33 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.170875161s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................XX.] duration:
733.628928ms target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:49:20 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 733.648976ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [........................................................XXX.] duration:
2.153405943s
FAIL (2025-09-18 16:49:14 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.153427554s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................XX.] duration:
13.285454688s
FAIL (2025-09-18 16:49:09 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 13.28547745s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.001177905s
FAIL (2025-09-18 16:48:56 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.001197221s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
30.001150749s
FAIL (2025-09-18 16:48:50 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.001163443s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.001110157s target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:48:37 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 30.001129693s
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.000622347s
FAIL (2025-09-18 16:48:20 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.000646212s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001054257s
FAIL (2025-09-18 16:47:49 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.001078011s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001082388s target:
https://ap1.storj.io/login
FAIL (2025-09-18 16:47:41 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 30.001103466s
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001230718s
FAIL (2025-09-18 16:47:29 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.001252217s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
792.630161ms
FAIL (2025-09-18 07:04:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 6)
======LOGS======
em: *pb.BatchRequestItem_SegmentBegin
Sep 18 06:21:00.406634509 (472h12m35.759419607s) [seg_pos={0 0}, upload=415651]: Begin upload segment...
Sep 18 06:21:00.406674804 (472h12m35.759459903s) [seg_pos={0 0}, upload=415651]: Done begin upload segment.
Sep 18 06:21:00.406696906 (472h12m35.759482014s) [noise=true , address=145.79.227.2:4113 , piece_id=74WZD4XJZLFYOLI65IL2EXYRG55I55R4GFIVC3SKF2QKVSHZSQOQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.406707956 (472h12m35.759493085s) [noise=true , address=45.91.116.6:4975 , piece_id=LGJYHBNENVYBYOJNZMR67ZKVF663BPVTDRKFRCHOL2RAVZG3NPCA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407029581 (472h12m35.759814689s) [noise=true , address=45.91.116.8:4047 , piece_id=HNMEIOAEK45NOSAJ574VIRGWILXHXSZY3VNTGPGJKKR77MAO757Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407042856 (472h12m35.759827954s) [noise=true , address=77.90.31.19:4841 , piece_id=3CKF4GT3D3RQ3O4KHADYF2FPPKKFISRV447YQM3YWYIKIY6LQC6Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407290631 (472h12m35.760075729s) [noise=true , address=45.91.116.5:4297 , piece_id=TR7NFAX6GDSFPVZZHTNLDO4SY56IE7JFAS4FODQ35KPO2FZXEC3Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407335154 (472h12m35.760120262s) [noise=true , address=77.90.31.12:4590 , piece_id=F3VBHMHF7RZEDSSOTW42TFMUD77NZW2LLUHEG5NQ4DOBIQG2FAXA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407534508 (472h12m35.760319616s) [noise=true , address=77.90.31.13:4628 , piece_id=5QY7TP5VCJUYHZ6INO5Y5UIXV6VS4OKQ3DAYKUWKLD4KYL2FZDGQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407643693 (472h12m35.760428791s) [noise=true , address=45.91.116.10:4082 , piece_id=DQ7BH6P3SXLXAIDWKUJ7ZLMPVU4QDDJ47U2EKVKRIRFHNZNC6NIQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407876960 (472h12m35.760662088s) [noise=true , address=145.79.227.15:4874 , piece_id=DZTMDVIJIYWHOUL4E4GGKSU6RZOTSIP5EQAD6ULRW5MPONZKHDFQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407976797 (472h12m35.760761896s) [noise=true , address=145.79.227.5:4221 , piece_id=WP7YXHNELCGBWJMU5XNKVSJNKJUYBXRMZU6OL2SGH4ADW37KNHYA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408191752 (472h12m35.760976849s) [noise=true , address=45.91.116.4:4937 , piece_id=NHZZ4RTJPLIRGOEZE35HMJM36UQNKZ7BBXANF3VOLR2L4NZIGOEA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408246144 (472h12m35.761031242s) [noise=true , address=45.91.116.14:4190 , piece_id=LAUZ7JI5CKTVCR5FYELFTVROKQU4AAW7KQEFYNKTS6OEVXQVIY5Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408443995 (472h12m35.761229103s) [noise=true , address=77.90.31.10:4517 , piece_id=KWZ7E5PTD2JYFFDA72AZOHSXL4ICSD2E2NQR3DPALIXOWQAKBPZA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408750219 (472h12m35.761535317s) [noise=true , address=45.91.116.11:4119 , piece_id=H3DPJDEBQZTTTFUJEPUMUXTUK3NRRXU644JFA6SKXRUSJ6PRXBDA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408995018 (472h12m35.761780116s) [noise=true , address=45.91.116.9:4262 , piece_id=ZT3MZQ6JJETFBP2UEPIOHNN6UA7HFRDYW6LTRAKRHA4B2YT42ILA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.409217326 (472h12m35.762002424s) [noise=true , address=145.79.227.16:4912 , piece_id=TWSWDK7RCN7RAFU2M4R2I4RMKWDHVAV242MGIBGUOZ5575SYV64A, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.482276928 (472h12m35.835062036s) [upload=415651]: Next returned nil segment
Sep 18 06:21:00.482298939 (472h12m35.835084047s) [upload=415651]: Waiting for error group managing segments...
Sep 18 06:21:00.494004563 (472h12m35.846789671s) [noise=true , address=145.79.227.2:4113 , piece_id=74WZD4XJZLFYOLI65IL2EXYRG55I55R4GFIVC3SKF2QKVSHZSQOQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:00.494305999 (472h12m35.847091098s) [noise=true , address=145.79.227.15:4874 , piece_id=DZTMDVIJIYWHOUL4E4GGKSU6RZOTSIP5EQAD6ULRW5MPONZKHDFQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:00.494521493 (472h12m35.847306601s) [noise=true , address=145.79.227.5:4221 , piece_id=WP7YXHNELCGBWJMU5XNKVSJNKJUYBXRMZU6OL2SGH4ADW37KNHYA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:00.495017263 (472h12m35.847802361s) [noise=true , address=145.79.227.16:4912 , piece_id=TWSWDK7RCN7RAFU2M4R2I4RMKWDHVAV242MGIBGUOZ5575SYV64A, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:29.335357246 (472h13m4.688142354s) [noise=true , address=45.91.116.4:4937 , piece_id=NHZZ4RTJPLIRGOEZE35HMJM36UQNKZ7BBXANF3VOLR2L4NZIGOEA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:30.078562165 (472h13m5.431347273s) [noise=true , address=45.91.116.14:4190 , piece_id=LAUZ7JI5CKTVCR5FYELFTVROKQU4AAW7KQEFYNKTS6OEVXQVIY5Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:30.373203149 (472h13m5.725988288s) [noise=true , address=77.90.31.13:4628 , piece_id=5QY7TP5VCJUYHZ6INO5Y5UIXV6VS4OKQ3DAYKUWKLD4KYL2FZDGQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373316822 (472h13m5.72610196s) [noise=true , address=77.90.31.12:4590 , piece_id=F3VBHMHF7RZEDSSOTW42TFMUD77NZW2LLUHEG5NQ4DOBIQG2FAXA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 5.161.89.252:17500->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373407473 (472h13m5.726192611s) [noise=true , address=45.91.116.8:4047 , piece_id=HNMEIOAEK45NOSAJ574VIRGWILXHXSZY3VNTGPGJKKR77MAO757Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 5.161.89.252:48174->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373436468 (472h13m5.726221596s) [noise=true , address=45.91.116.9:4262 , piece_id=ZT3MZQ6JJETFBP2UEPIOHNN6UA7HFRDYW6LTRAKRHA4B2YT42ILA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373493525 (472h13m5.726278653s) [noise=true , address=45.91.116.10:4082 , piece_id=DQ7BH6P3SXLXAIDWKUJ7ZLMPVU4QDDJ47U2EKVKRIRFHNZNC6NIQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373503924 (472h13m5.726289053s) [noise=true , address=45.91.116.6:4975 , piece_id=LGJYHBNENVYBYOJNZMR67ZKVF663BPVTDRKFRCHOL2RAVZG3NPCA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: rpc: dial tcp 45.91.116.6:4975: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373594234 (472h13m5.726379362s) [noise=true , address=77.90.31.19:4841 , piece_id=3CKF4GT3D3RQ3O4KHADYF2FPPKKFISRV447YQM3YWYIKIY6LQC6Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373642995 (472h13m5.726428123s) [noise=true , address=45.91.116.11:4119 , piece_id=H3DPJDEBQZTTTFUJEPUMUXTUK3NRRXU644JFA6SKXRUSJ6PRXBDA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 5.161.89.252:46646->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373655599 (472h13m5.726440737s) [noise=true , address=77.90.31.10:4517 , piece_id=KWZ7E5PTD2JYFFDA72AZOHSXL4ICSD2E2NQR3DPALIXOWQAKBPZA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373716553 (472h13m5.726501681s) [noise=true , address=45.91.116.5:4297 , piece_id=TR7NFAX6GDSFPVZZHTNLDO4SY56IE7JFAS4FODQ35KPO2FZXEC3Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: read tcp 5.161.89.252:12364->45.91.116.5:4297: use of closed network connection; context deadline exceeded
Sep 18 06:21:30.373748693 (472h13m5.726533832s) [seg_pos={0 0}, upload=415651]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil>
Sep 18 06:21:30.388943886 (472h13m5.741729014s) [upload=415651]: Done uploading segments...
Sep 18 06:21:30.388973221 (472h13m5.741758349s) [upload=415651]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 792.645809ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
673.434129ms
FAIL (2025-09-18 07:03:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 6)
======LOGS======
em: *pb.BatchRequestItem_SegmentBegin
Sep 18 06:21:00.406634509 (472h12m35.759419607s) [seg_pos={0 0}, upload=415651]: Begin upload segment...
Sep 18 06:21:00.406674804 (472h12m35.759459903s) [seg_pos={0 0}, upload=415651]: Done begin upload segment.
Sep 18 06:21:00.406696906 (472h12m35.759482014s) [noise=true , address=145.79.227.2:4113 , piece_id=74WZD4XJZLFYOLI65IL2EXYRG55I55R4GFIVC3SKF2QKVSHZSQOQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.406707956 (472h12m35.759493085s) [noise=true , address=45.91.116.6:4975 , piece_id=LGJYHBNENVYBYOJNZMR67ZKVF663BPVTDRKFRCHOL2RAVZG3NPCA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407029581 (472h12m35.759814689s) [noise=true , address=45.91.116.8:4047 , piece_id=HNMEIOAEK45NOSAJ574VIRGWILXHXSZY3VNTGPGJKKR77MAO757Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407042856 (472h12m35.759827954s) [noise=true , address=77.90.31.19:4841 , piece_id=3CKF4GT3D3RQ3O4KHADYF2FPPKKFISRV447YQM3YWYIKIY6LQC6Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407290631 (472h12m35.760075729s) [noise=true , address=45.91.116.5:4297 , piece_id=TR7NFAX6GDSFPVZZHTNLDO4SY56IE7JFAS4FODQ35KPO2FZXEC3Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407335154 (472h12m35.760120262s) [noise=true , address=77.90.31.12:4590 , piece_id=F3VBHMHF7RZEDSSOTW42TFMUD77NZW2LLUHEG5NQ4DOBIQG2FAXA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407534508 (472h12m35.760319616s) [noise=true , address=77.90.31.13:4628 , piece_id=5QY7TP5VCJUYHZ6INO5Y5UIXV6VS4OKQ3DAYKUWKLD4KYL2FZDGQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407643693 (472h12m35.760428791s) [noise=true , address=45.91.116.10:4082 , piece_id=DQ7BH6P3SXLXAIDWKUJ7ZLMPVU4QDDJ47U2EKVKRIRFHNZNC6NIQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407876960 (472h12m35.760662088s) [noise=true , address=145.79.227.15:4874 , piece_id=DZTMDVIJIYWHOUL4E4GGKSU6RZOTSIP5EQAD6ULRW5MPONZKHDFQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407976797 (472h12m35.760761896s) [noise=true , address=145.79.227.5:4221 , piece_id=WP7YXHNELCGBWJMU5XNKVSJNKJUYBXRMZU6OL2SGH4ADW37KNHYA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408191752 (472h12m35.760976849s) [noise=true , address=45.91.116.4:4937 , piece_id=NHZZ4RTJPLIRGOEZE35HMJM36UQNKZ7BBXANF3VOLR2L4NZIGOEA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408246144 (472h12m35.761031242s) [noise=true , address=45.91.116.14:4190 , piece_id=LAUZ7JI5CKTVCR5FYELFTVROKQU4AAW7KQEFYNKTS6OEVXQVIY5Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408443995 (472h12m35.761229103s) [noise=true , address=77.90.31.10:4517 , piece_id=KWZ7E5PTD2JYFFDA72AZOHSXL4ICSD2E2NQR3DPALIXOWQAKBPZA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408750219 (472h12m35.761535317s) [noise=true , address=45.91.116.11:4119 , piece_id=H3DPJDEBQZTTTFUJEPUMUXTUK3NRRXU644JFA6SKXRUSJ6PRXBDA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408995018 (472h12m35.761780116s) [noise=true , address=45.91.116.9:4262 , piece_id=ZT3MZQ6JJETFBP2UEPIOHNN6UA7HFRDYW6LTRAKRHA4B2YT42ILA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.409217326 (472h12m35.762002424s) [noise=true , address=145.79.227.16:4912 , piece_id=TWSWDK7RCN7RAFU2M4R2I4RMKWDHVAV242MGIBGUOZ5575SYV64A, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.482276928 (472h12m35.835062036s) [upload=415651]: Next returned nil segment
Sep 18 06:21:00.482298939 (472h12m35.835084047s) [upload=415651]: Waiting for error group managing segments...
Sep 18 06:21:00.494004563 (472h12m35.846789671s) [noise=true , address=145.79.227.2:4113 , piece_id=74WZD4XJZLFYOLI65IL2EXYRG55I55R4GFIVC3SKF2QKVSHZSQOQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:00.494305999 (472h12m35.847091098s) [noise=true , address=145.79.227.15:4874 , piece_id=DZTMDVIJIYWHOUL4E4GGKSU6RZOTSIP5EQAD6ULRW5MPONZKHDFQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:00.494521493 (472h12m35.847306601s) [noise=true , address=145.79.227.5:4221 , piece_id=WP7YXHNELCGBWJMU5XNKVSJNKJUYBXRMZU6OL2SGH4ADW37KNHYA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:00.495017263 (472h12m35.847802361s) [noise=true , address=145.79.227.16:4912 , piece_id=TWSWDK7RCN7RAFU2M4R2I4RMKWDHVAV242MGIBGUOZ5575SYV64A, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:29.335357246 (472h13m4.688142354s) [noise=true , address=45.91.116.4:4937 , piece_id=NHZZ4RTJPLIRGOEZE35HMJM36UQNKZ7BBXANF3VOLR2L4NZIGOEA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:30.078562165 (472h13m5.431347273s) [noise=true , address=45.91.116.14:4190 , piece_id=LAUZ7JI5CKTVCR5FYELFTVROKQU4AAW7KQEFYNKTS6OEVXQVIY5Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:30.373203149 (472h13m5.725988288s) [noise=true , address=77.90.31.13:4628 , piece_id=5QY7TP5VCJUYHZ6INO5Y5UIXV6VS4OKQ3DAYKUWKLD4KYL2FZDGQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373316822 (472h13m5.72610196s) [noise=true , address=77.90.31.12:4590 , piece_id=F3VBHMHF7RZEDSSOTW42TFMUD77NZW2LLUHEG5NQ4DOBIQG2FAXA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 5.161.89.252:17500->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373407473 (472h13m5.726192611s) [noise=true , address=45.91.116.8:4047 , piece_id=HNMEIOAEK45NOSAJ574VIRGWILXHXSZY3VNTGPGJKKR77MAO757Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 5.161.89.252:48174->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373436468 (472h13m5.726221596s) [noise=true , address=45.91.116.9:4262 , piece_id=ZT3MZQ6JJETFBP2UEPIOHNN6UA7HFRDYW6LTRAKRHA4B2YT42ILA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373493525 (472h13m5.726278653s) [noise=true , address=45.91.116.10:4082 , piece_id=DQ7BH6P3SXLXAIDWKUJ7ZLMPVU4QDDJ47U2EKVKRIRFHNZNC6NIQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373503924 (472h13m5.726289053s) [noise=true , address=45.91.116.6:4975 , piece_id=LGJYHBNENVYBYOJNZMR67ZKVF663BPVTDRKFRCHOL2RAVZG3NPCA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: rpc: dial tcp 45.91.116.6:4975: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373594234 (472h13m5.726379362s) [noise=true , address=77.90.31.19:4841 , piece_id=3CKF4GT3D3RQ3O4KHADYF2FPPKKFISRV447YQM3YWYIKIY6LQC6Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373642995 (472h13m5.726428123s) [noise=true , address=45.91.116.11:4119 , piece_id=H3DPJDEBQZTTTFUJEPUMUXTUK3NRRXU644JFA6SKXRUSJ6PRXBDA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 5.161.89.252:46646->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373655599 (472h13m5.726440737s) [noise=true , address=77.90.31.10:4517 , piece_id=KWZ7E5PTD2JYFFDA72AZOHSXL4ICSD2E2NQR3DPALIXOWQAKBPZA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373716553 (472h13m5.726501681s) [noise=true , address=45.91.116.5:4297 , piece_id=TR7NFAX6GDSFPVZZHTNLDO4SY56IE7JFAS4FODQ35KPO2FZXEC3Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: read tcp 5.161.89.252:12364->45.91.116.5:4297: use of closed network connection; context deadline exceeded
Sep 18 06:21:30.373748693 (472h13m5.726533832s) [seg_pos={0 0}, upload=415651]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil>
Sep 18 06:21:30.388943886 (472h13m5.741729014s) [upload=415651]: Done uploading segments...
Sep 18 06:21:30.388973221 (472h13m5.741758349s) [upload=415651]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 673.464836ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
511.182366ms
FAIL (2025-09-18 07:02:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 511.21126ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
640.791948ms
FAIL (2025-09-18 07:02:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 6)
======LOGS======
em: *pb.BatchRequestItem_SegmentBegin
Sep 18 06:21:00.406634509 (472h12m35.759419607s) [seg_pos={0 0}, upload=415651]: Begin upload segment...
Sep 18 06:21:00.406674804 (472h12m35.759459903s) [seg_pos={0 0}, upload=415651]: Done begin upload segment.
Sep 18 06:21:00.406696906 (472h12m35.759482014s) [noise=true , address=145.79.227.2:4113 , piece_id=74WZD4XJZLFYOLI65IL2EXYRG55I55R4GFIVC3SKF2QKVSHZSQOQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.406707956 (472h12m35.759493085s) [noise=true , address=45.91.116.6:4975 , piece_id=LGJYHBNENVYBYOJNZMR67ZKVF663BPVTDRKFRCHOL2RAVZG3NPCA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407029581 (472h12m35.759814689s) [noise=true , address=45.91.116.8:4047 , piece_id=HNMEIOAEK45NOSAJ574VIRGWILXHXSZY3VNTGPGJKKR77MAO757Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407042856 (472h12m35.759827954s) [noise=true , address=77.90.31.19:4841 , piece_id=3CKF4GT3D3RQ3O4KHADYF2FPPKKFISRV447YQM3YWYIKIY6LQC6Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407290631 (472h12m35.760075729s) [noise=true , address=45.91.116.5:4297 , piece_id=TR7NFAX6GDSFPVZZHTNLDO4SY56IE7JFAS4FODQ35KPO2FZXEC3Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407335154 (472h12m35.760120262s) [noise=true , address=77.90.31.12:4590 , piece_id=F3VBHMHF7RZEDSSOTW42TFMUD77NZW2LLUHEG5NQ4DOBIQG2FAXA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407534508 (472h12m35.760319616s) [noise=true , address=77.90.31.13:4628 , piece_id=5QY7TP5VCJUYHZ6INO5Y5UIXV6VS4OKQ3DAYKUWKLD4KYL2FZDGQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407643693 (472h12m35.760428791s) [noise=true , address=45.91.116.10:4082 , piece_id=DQ7BH6P3SXLXAIDWKUJ7ZLMPVU4QDDJ47U2EKVKRIRFHNZNC6NIQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407876960 (472h12m35.760662088s) [noise=true , address=145.79.227.15:4874 , piece_id=DZTMDVIJIYWHOUL4E4GGKSU6RZOTSIP5EQAD6ULRW5MPONZKHDFQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407976797 (472h12m35.760761896s) [noise=true , address=145.79.227.5:4221 , piece_id=WP7YXHNELCGBWJMU5XNKVSJNKJUYBXRMZU6OL2SGH4ADW37KNHYA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408191752 (472h12m35.760976849s) [noise=true , address=45.91.116.4:4937 , piece_id=NHZZ4RTJPLIRGOEZE35HMJM36UQNKZ7BBXANF3VOLR2L4NZIGOEA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408246144 (472h12m35.761031242s) [noise=true , address=45.91.116.14:4190 , piece_id=LAUZ7JI5CKTVCR5FYELFTVROKQU4AAW7KQEFYNKTS6OEVXQVIY5Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408443995 (472h12m35.761229103s) [noise=true , address=77.90.31.10:4517 , piece_id=KWZ7E5PTD2JYFFDA72AZOHSXL4ICSD2E2NQR3DPALIXOWQAKBPZA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408750219 (472h12m35.761535317s) [noise=true , address=45.91.116.11:4119 , piece_id=H3DPJDEBQZTTTFUJEPUMUXTUK3NRRXU644JFA6SKXRUSJ6PRXBDA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408995018 (472h12m35.761780116s) [noise=true , address=45.91.116.9:4262 , piece_id=ZT3MZQ6JJETFBP2UEPIOHNN6UA7HFRDYW6LTRAKRHA4B2YT42ILA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.409217326 (472h12m35.762002424s) [noise=true , address=145.79.227.16:4912 , piece_id=TWSWDK7RCN7RAFU2M4R2I4RMKWDHVAV242MGIBGUOZ5575SYV64A, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.482276928 (472h12m35.835062036s) [upload=415651]: Next returned nil segment
Sep 18 06:21:00.482298939 (472h12m35.835084047s) [upload=415651]: Waiting for error group managing segments...
Sep 18 06:21:00.494004563 (472h12m35.846789671s) [noise=true , address=145.79.227.2:4113 , piece_id=74WZD4XJZLFYOLI65IL2EXYRG55I55R4GFIVC3SKF2QKVSHZSQOQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:00.494305999 (472h12m35.847091098s) [noise=true , address=145.79.227.15:4874 , piece_id=DZTMDVIJIYWHOUL4E4GGKSU6RZOTSIP5EQAD6ULRW5MPONZKHDFQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:00.494521493 (472h12m35.847306601s) [noise=true , address=145.79.227.5:4221 , piece_id=WP7YXHNELCGBWJMU5XNKVSJNKJUYBXRMZU6OL2SGH4ADW37KNHYA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:00.495017263 (472h12m35.847802361s) [noise=true , address=145.79.227.16:4912 , piece_id=TWSWDK7RCN7RAFU2M4R2I4RMKWDHVAV242MGIBGUOZ5575SYV64A, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:29.335357246 (472h13m4.688142354s) [noise=true , address=45.91.116.4:4937 , piece_id=NHZZ4RTJPLIRGOEZE35HMJM36UQNKZ7BBXANF3VOLR2L4NZIGOEA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:30.078562165 (472h13m5.431347273s) [noise=true , address=45.91.116.14:4190 , piece_id=LAUZ7JI5CKTVCR5FYELFTVROKQU4AAW7KQEFYNKTS6OEVXQVIY5Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:30.373203149 (472h13m5.725988288s) [noise=true , address=77.90.31.13:4628 , piece_id=5QY7TP5VCJUYHZ6INO5Y5UIXV6VS4OKQ3DAYKUWKLD4KYL2FZDGQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373316822 (472h13m5.72610196s) [noise=true , address=77.90.31.12:4590 , piece_id=F3VBHMHF7RZEDSSOTW42TFMUD77NZW2LLUHEG5NQ4DOBIQG2FAXA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 5.161.89.252:17500->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373407473 (472h13m5.726192611s) [noise=true , address=45.91.116.8:4047 , piece_id=HNMEIOAEK45NOSAJ574VIRGWILXHXSZY3VNTGPGJKKR77MAO757Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 5.161.89.252:48174->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373436468 (472h13m5.726221596s) [noise=true , address=45.91.116.9:4262 , piece_id=ZT3MZQ6JJETFBP2UEPIOHNN6UA7HFRDYW6LTRAKRHA4B2YT42ILA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373493525 (472h13m5.726278653s) [noise=true , address=45.91.116.10:4082 , piece_id=DQ7BH6P3SXLXAIDWKUJ7ZLMPVU4QDDJ47U2EKVKRIRFHNZNC6NIQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373503924 (472h13m5.726289053s) [noise=true , address=45.91.116.6:4975 , piece_id=LGJYHBNENVYBYOJNZMR67ZKVF663BPVTDRKFRCHOL2RAVZG3NPCA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: rpc: dial tcp 45.91.116.6:4975: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373594234 (472h13m5.726379362s) [noise=true , address=77.90.31.19:4841 , piece_id=3CKF4GT3D3RQ3O4KHADYF2FPPKKFISRV447YQM3YWYIKIY6LQC6Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373642995 (472h13m5.726428123s) [noise=true , address=45.91.116.11:4119 , piece_id=H3DPJDEBQZTTTFUJEPUMUXTUK3NRRXU644JFA6SKXRUSJ6PRXBDA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 5.161.89.252:46646->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373655599 (472h13m5.726440737s) [noise=true , address=77.90.31.10:4517 , piece_id=KWZ7E5PTD2JYFFDA72AZOHSXL4ICSD2E2NQR3DPALIXOWQAKBPZA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373716553 (472h13m5.726501681s) [noise=true , address=45.91.116.5:4297 , piece_id=TR7NFAX6GDSFPVZZHTNLDO4SY56IE7JFAS4FODQ35KPO2FZXEC3Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: read tcp 5.161.89.252:12364->45.91.116.5:4297: use of closed network connection; context deadline exceeded
Sep 18 06:21:30.373748693 (472h13m5.726533832s) [seg_pos={0 0}, upload=415651]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil>
Sep 18 06:21:30.388943886 (472h13m5.741729014s) [upload=415651]: Done uploading segments...
Sep 18 06:21:30.388973221 (472h13m5.741758349s) [upload=415651]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 640.808398ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
458.129695ms
FAIL (2025-09-18 07:02:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 458.148049ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
885.708326ms
FAIL (2025-09-18 07:01:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 885.729305ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
820.828853ms
FAIL (2025-09-18 07:01:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 6)
======LOGS======
em: *pb.BatchRequestItem_SegmentBegin
Sep 18 06:21:00.406634509 (472h12m35.759419607s) [seg_pos={0 0}, upload=415651]: Begin upload segment...
Sep 18 06:21:00.406674804 (472h12m35.759459903s) [seg_pos={0 0}, upload=415651]: Done begin upload segment.
Sep 18 06:21:00.406696906 (472h12m35.759482014s) [noise=true , address=145.79.227.2:4113 , piece_id=74WZD4XJZLFYOLI65IL2EXYRG55I55R4GFIVC3SKF2QKVSHZSQOQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.406707956 (472h12m35.759493085s) [noise=true , address=45.91.116.6:4975 , piece_id=LGJYHBNENVYBYOJNZMR67ZKVF663BPVTDRKFRCHOL2RAVZG3NPCA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407029581 (472h12m35.759814689s) [noise=true , address=45.91.116.8:4047 , piece_id=HNMEIOAEK45NOSAJ574VIRGWILXHXSZY3VNTGPGJKKR77MAO757Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407042856 (472h12m35.759827954s) [noise=true , address=77.90.31.19:4841 , piece_id=3CKF4GT3D3RQ3O4KHADYF2FPPKKFISRV447YQM3YWYIKIY6LQC6Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407290631 (472h12m35.760075729s) [noise=true , address=45.91.116.5:4297 , piece_id=TR7NFAX6GDSFPVZZHTNLDO4SY56IE7JFAS4FODQ35KPO2FZXEC3Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407335154 (472h12m35.760120262s) [noise=true , address=77.90.31.12:4590 , piece_id=F3VBHMHF7RZEDSSOTW42TFMUD77NZW2LLUHEG5NQ4DOBIQG2FAXA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407534508 (472h12m35.760319616s) [noise=true , address=77.90.31.13:4628 , piece_id=5QY7TP5VCJUYHZ6INO5Y5UIXV6VS4OKQ3DAYKUWKLD4KYL2FZDGQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407643693 (472h12m35.760428791s) [noise=true , address=45.91.116.10:4082 , piece_id=DQ7BH6P3SXLXAIDWKUJ7ZLMPVU4QDDJ47U2EKVKRIRFHNZNC6NIQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407876960 (472h12m35.760662088s) [noise=true , address=145.79.227.15:4874 , piece_id=DZTMDVIJIYWHOUL4E4GGKSU6RZOTSIP5EQAD6ULRW5MPONZKHDFQ, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.407976797 (472h12m35.760761896s) [noise=true , address=145.79.227.5:4221 , piece_id=WP7YXHNELCGBWJMU5XNKVSJNKJUYBXRMZU6OL2SGH4ADW37KNHYA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408191752 (472h12m35.760976849s) [noise=true , address=45.91.116.4:4937 , piece_id=NHZZ4RTJPLIRGOEZE35HMJM36UQNKZ7BBXANF3VOLR2L4NZIGOEA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408246144 (472h12m35.761031242s) [noise=true , address=45.91.116.14:4190 , piece_id=LAUZ7JI5CKTVCR5FYELFTVROKQU4AAW7KQEFYNKTS6OEVXQVIY5Q, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408443995 (472h12m35.761229103s) [noise=true , address=77.90.31.10:4517 , piece_id=KWZ7E5PTD2JYFFDA72AZOHSXL4ICSD2E2NQR3DPALIXOWQAKBPZA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408750219 (472h12m35.761535317s) [noise=true , address=45.91.116.11:4119 , piece_id=H3DPJDEBQZTTTFUJEPUMUXTUK3NRRXU644JFA6SKXRUSJ6PRXBDA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.408995018 (472h12m35.761780116s) [noise=true , address=45.91.116.9:4262 , piece_id=ZT3MZQ6JJETFBP2UEPIOHNN6UA7HFRDYW6LTRAKRHA4B2YT42ILA, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.409217326 (472h12m35.762002424s) [noise=true , address=145.79.227.16:4912 , piece_id=TWSWDK7RCN7RAFU2M4R2I4RMKWDHVAV242MGIBGUOZ5575SYV64A, seg_pos={0 0}, upload=415651]: Uploading piece...
Sep 18 06:21:00.482276928 (472h12m35.835062036s) [upload=415651]: Next returned nil segment
Sep 18 06:21:00.482298939 (472h12m35.835084047s) [upload=415651]: Waiting for error group managing segments...
Sep 18 06:21:00.494004563 (472h12m35.846789671s) [noise=true , address=145.79.227.2:4113 , piece_id=74WZD4XJZLFYOLI65IL2EXYRG55I55R4GFIVC3SKF2QKVSHZSQOQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:00.494305999 (472h12m35.847091098s) [noise=true , address=145.79.227.15:4874 , piece_id=DZTMDVIJIYWHOUL4E4GGKSU6RZOTSIP5EQAD6ULRW5MPONZKHDFQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:00.494521493 (472h12m35.847306601s) [noise=true , address=145.79.227.5:4221 , piece_id=WP7YXHNELCGBWJMU5XNKVSJNKJUYBXRMZU6OL2SGH4ADW37KNHYA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:00.495017263 (472h12m35.847802361s) [noise=true , address=145.79.227.16:4912 , piece_id=TWSWDK7RCN7RAFU2M4R2I4RMKWDHVAV242MGIBGUOZ5575SYV64A, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:29.335357246 (472h13m4.688142354s) [noise=true , address=45.91.116.4:4937 , piece_id=NHZZ4RTJPLIRGOEZE35HMJM36UQNKZ7BBXANF3VOLR2L4NZIGOEA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:30.078562165 (472h13m5.431347273s) [noise=true , address=45.91.116.14:4190 , piece_id=LAUZ7JI5CKTVCR5FYELFTVROKQU4AAW7KQEFYNKTS6OEVXQVIY5Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: <nil>
Sep 18 06:21:30.373203149 (472h13m5.725988288s) [noise=true , address=77.90.31.13:4628 , piece_id=5QY7TP5VCJUYHZ6INO5Y5UIXV6VS4OKQ3DAYKUWKLD4KYL2FZDGQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373316822 (472h13m5.72610196s) [noise=true , address=77.90.31.12:4590 , piece_id=F3VBHMHF7RZEDSSOTW42TFMUD77NZW2LLUHEG5NQ4DOBIQG2FAXA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 5.161.89.252:17500->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373407473 (472h13m5.726192611s) [noise=true , address=45.91.116.8:4047 , piece_id=HNMEIOAEK45NOSAJ574VIRGWILXHXSZY3VNTGPGJKKR77MAO757Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 5.161.89.252:48174->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373436468 (472h13m5.726221596s) [noise=true , address=45.91.116.9:4262 , piece_id=ZT3MZQ6JJETFBP2UEPIOHNN6UA7HFRDYW6LTRAKRHA4B2YT42ILA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373493525 (472h13m5.726278653s) [noise=true , address=45.91.116.10:4082 , piece_id=DQ7BH6P3SXLXAIDWKUJ7ZLMPVU4QDDJ47U2EKVKRIRFHNZNC6NIQ, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373503924 (472h13m5.726289053s) [noise=true , address=45.91.116.6:4975 , piece_id=LGJYHBNENVYBYOJNZMR67ZKVF663BPVTDRKFRCHOL2RAVZG3NPCA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: rpc: dial tcp 45.91.116.6:4975: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373594234 (472h13m5.726379362s) [noise=true , address=77.90.31.19:4841 , piece_id=3CKF4GT3D3RQ3O4KHADYF2FPPKKFISRV447YQM3YWYIKIY6LQC6Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: rpc: dial tcp 77.90.31.19:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373642995 (472h13m5.726428123s) [noise=true , address=45.91.116.11:4119 , piece_id=H3DPJDEBQZTTTFUJEPUMUXTUK3NRRXU644JFA6SKXRUSJ6PRXBDA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 5.161.89.252:46646->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373655599 (472h13m5.726440737s) [noise=true , address=77.90.31.10:4517 , piece_id=KWZ7E5PTD2JYFFDA72AZOHSXL4ICSD2E2NQR3DPALIXOWQAKBPZA, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 18 06:21:30.373716553 (472h13m5.726501681s) [noise=true , address=45.91.116.5:4297 , piece_id=TR7NFAX6GDSFPVZZHTNLDO4SY56IE7JFAS4FODQ35KPO2FZXEC3Q, seg_pos={0 0}, upload=415651]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: read tcp 5.161.89.252:12364->45.91.116.5:4297: use of closed network connection; context deadline exceeded
Sep 18 06:21:30.373748693 (472h13m5.726533832s) [seg_pos={0 0}, upload=415651]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil>
Sep 18 06:21:30.388943886 (472h13m5.741729014s) [upload=415651]: Done uploading segments...
Sep 18 06:21:30.388973221 (472h13m5.741758349s) [upload=415651]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 820.848501ms
core
uplink
us-select-1
|