Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
176.457018ms
FAIL (2025-08-31 10:44:32 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 176.474711ms
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................XX...............] duration:
2.873612757s
FAIL (2025-08-31 10:44:30 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.873635188s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........................................XX.XXX...........] duration:
499.005851ms
FAIL (2025-08-31 10:44:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
g 31 10:36:12.150915169 (44h27m47.503700297s) [upload=39171]: Done starting upload
Aug 31 10:36:12.151058448 (44h27m47.503843586s) [upload=39171]: Uploading segments...
Aug 31 10:36:12.151350147 (44h27m47.504135276s) [upload=39171]: Got next segment. Inline: false
Aug 31 10:36:12.151407877 (44h27m47.504193015s) [upload=39171]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 10:36:12.151414058 (44h27m47.504199186s) [upload=39171]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 10:36:12.181724654 (44h27m47.534509782s) [seg_pos={0 0}, upload=39171]: Begin upload segment...
Aug 31 10:36:12.181759048 (44h27m47.534544176s) [seg_pos={0 0}, upload=39171]: Done begin upload segment.
Aug 31 10:36:12.181787412 (44h27m47.534572551s) [noise=true , address=104.36.82.47:4543 , piece_id=OKISSUYRQP4GGIRDDPYFO6MCHDAFJEWCGHLI3P5T7EHBGOD7CYQA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.181865228 (44h27m47.534650367s) [noise=true , address=45.91.116.12:4239 , piece_id=JIN2FAKX4JF3WVYMMICMIFQDIJT7ZZ4XCVCTDTKB7B3OFDTBY6HQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182335613 (44h27m47.535120752s) [noise=true , address=77.90.31.10:4545 , piece_id=R5HG7NAZ2BFJCIYJOMV4BAD3Q7MAANXZJIFUDHBJ56PMITAB7FJA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182439810 (44h27m47.535224938s) [noise=true , address=104.36.82.11:4130 , piece_id=NEJRZC6MR7DTWS62XZZPJXTZW6GVHK4HIK6BXFD3QI46UGQACR3Q, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182733102 (44h27m47.53551823s) [noise=true , address=45.91.116.7:4034 , piece_id=Z5BA6X346T3H2CITKWTLJDZOUOTVCHTXPUDPO2OKKXGRQF3YUMKQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182828772 (44h27m47.535613901s) [noise=true , address=79.127.205.246:4280 , piece_id=BG3RNCBTBDNRY3MMJUQNKJP2RWH57UODQBKK66KK2WEDXA6FJQWA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183134137 (44h27m47.535919276s) [noise=true , address=104.36.82.48:4589 , piece_id=U4VSTDANJ6ZGA6PTOLZBKGBLD6K73SG4FQT6PUIH3QPWIO5A7YQQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183571981 (44h27m47.53635711s) [noise=true , address=77.90.31.13:4629 , piece_id=746YIFKZ7XTJCLHAV2K2TWYRIJP5PFS47B5D37PFT6MERB62QM6Q, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183905159 (44h27m47.536690288s) [noise=true , address=104.36.82.55:4212 , piece_id=BBTOTIURLHZXSY3NUBOJXHOB4U3HJIFEJM2AUO35DFACNAWBEZJA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184080729 (44h27m47.536865858s) [noise=true , address=45.91.116.13:4176 , piece_id=TTOEVZ6NGPERE27QNANZQVNP2QSCVGITHW5EFOV34EJ2KJ2GTVQA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184383600 (44h27m47.537168737s) [noise=true , address=77.90.31.17:4787 , piece_id=YLZWY3HQRSKHUHBOOVY36TATZ5IN7X2C27IXXCJKZZF4MHSRYH4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184492785 (44h27m47.537277914s) [noise=true , address=77.90.31.14:4678 , piece_id=4XGVSFNNY6MGP5IOHJIBXL3S7JIPYAXKHFIAE6BSX5TAKTCJ4JVQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184744750 (44h27m47.537529878s) [noise=true , address=45.91.116.9:4281 , piece_id=EWWXWQPBEDE724UFU2Z7YJ4E5DUI45OMNLHEFCC4OMG7ZQLWWBBQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184850529 (44h27m47.537635657s) [noise=true , address=104.36.82.56:4264 , piece_id=S67WK5JPS46KPUHBQPLDDRGYJOLN6EEVOE5BFYT74U2IIXZFWXTA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185105669 (44h27m47.537890807s) [noise=true , address=77.90.31.21:4891 , piece_id=IKWWTL4THNTJGQ3CQLADO4KY2KFOXPASHSALNGE7NBUAXHPV3MBA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185259740 (44h27m47.538044868s) [noise=true , address=64.49.13.138:4482 , piece_id=JNYXEQ6PTZYSW5ULU55W5ZFEBEW25KXCDA36K47K4ADKODUZOKYA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185545127 (44h27m47.538330255s) [noise=true , address=104.36.82.23:4458 , piece_id=Y5IELCEOPOLHYYU2WEAZM7ZQMS3W6NR5YS64Y4UYD5MYZZQUJY4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185657418 (44h27m47.538442556s) [noise=true , address=104.36.82.44:4883 , piece_id=455DKBRQXXYON4JM5CQ4SE5WR3WTTH3AJUAB6ZEHEHSE4RE6LP4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.220474138 (44h27m47.573259266s) [noise=true , address=77.90.31.21:4891 , piece_id=IKWWTL4THNTJGQ3CQLADO4KY2KFOXPASHSALNGE7NBUAXHPV3MBA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: write tcp 5.161.89.252:27554->77.90.31.21:4891: write: no route to host; conn closed: op
Aug 31 10:36:12.302528983 (44h27m47.655314111s) [upload=39171]: Next returned nil segment
Aug 31 10:36:12.302541757 (44h27m47.655326885s) [upload=39171]: Waiting for error group managing segments...
Aug 31 10:36:12.316638798 (44h27m47.669423926s) [noise=true , address=104.36.82.56:4264 , piece_id=S67WK5JPS46KPUHBQPLDDRGYJOLN6EEVOE5BFYT74U2IIXZFWXTA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.316773852 (44h27m47.66955899s) [noise=true , address=104.36.82.44:4883 , piece_id=455DKBRQXXYON4JM5CQ4SE5WR3WTTH3AJUAB6ZEHEHSE4RE6LP4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317388379 (44h27m47.670173506s) [noise=true , address=104.36.82.47:4543 , piece_id=OKISSUYRQP4GGIRDDPYFO6MCHDAFJEWCGHLI3P5T7EHBGOD7CYQA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317831102 (44h27m47.67061623s) [noise=true , address=104.36.82.23:4458 , piece_id=Y5IELCEOPOLHYYU2WEAZM7ZQMS3W6NR5YS64Y4UYD5MYZZQUJY4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317981115 (44h27m47.670766243s) [noise=true , address=104.36.82.55:4212 , piece_id=BBTOTIURLHZXSY3NUBOJXHOB4U3HJIFEJM2AUO35DFACNAWBEZJA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.318644403 (44h27m47.671429531s) [noise=true , address=104.36.82.11:4130 , piece_id=NEJRZC6MR7DTWS62XZZPJXTZW6GVHK4HIK6BXFD3QI46UGQACR3Q, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.319463665 (44h27m47.672248794s) [noise=true , address=104.36.82.48:4589 , piece_id=U4VSTDANJ6ZGA6PTOLZBKGBLD6K73SG4FQT6PUIH3QPWIO5A7YQQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.365155230 (44h27m47.717940358s) [noise=true , address=45.91.116.13:4176 , piece_id=TTOEVZ6NGPERE27QNANZQVNP2QSCVGITHW5EFOV34EJ2KJ2GTVQA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.382100673 (44h27m47.734885801s) [noise=true , address=45.91.116.9:4281 , piece_id=EWWXWQPBEDE724UFU2Z7YJ4E5DUI45OMNLHEFCC4OMG7ZQLWWBBQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.382477341 (44h27m47.73526248s) [noise=true , address=45.91.116.12:4239 , piece_id=JIN2FAKX4JF3WVYMMICMIFQDIJT7ZZ4XCVCTDTKB7B3OFDTBY6HQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.385851305 (44h27m47.738636442s) [noise=true , address=45.91.116.7:4034 , piece_id=Z5BA6X346T3H2CITKWTLJDZOUOTVCHTXPUDPO2OKKXGRQF3YUMKQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.670547674 (44h27m48.023332772s) [noise=true , address=79.127.205.246:4280 , piece_id=BG3RNCBTBDNRY3MMJUQNKJP2RWH57UODQBKK66KK2WEDXA6FJQWA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.734232925 (44h27m48.087018034s) [noise=true , address=64.49.13.138:4482 , piece_id=JNYXEQ6PTZYSW5ULU55W5ZFEBEW25KXCDA36K47K4ADKODUZOKYA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:37:12.136109118 (44h28m47.488894215s) [noise=true , address=77.90.31.17:4787 , piece_id=YLZWY3HQRSKHUHBOOVY36TATZ5IN7X2C27IXXCJKZZF4MHSRYH4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: rpc: dial tcp 77.90.31.17:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136180632 (44h28m47.48896573s) [noise=true , address=77.90.31.10:4545 , piece_id=R5HG7NAZ2BFJCIYJOMV4BAD3Q7MAANXZJIFUDHBJ56PMITAB7FJA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:1ZFFijCUFET1ShEJuav6uAzFhfraCEcr6seuySYN9ipn4wsZGX, address:77.90.31.10:4545): protocol: rpc: dial tcp 77.90.31.10:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136217150 (44h28m47.489002248s) [noise=true , address=77.90.31.13:4629 , piece_id=746YIFKZ7XTJCLHAV2K2TWYRIJP5PFS47B5D37PFT6MERB62QM6Q, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136244112 (44h28m47.48902921s) [noise=true , address=77.90.31.14:4678 , piece_id=4XGVSFNNY6MGP5IOHJIBXL3S7JIPYAXKHFIAE6BSX5TAKTCJ4JVQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136292463 (44h28m47.48907756s) [seg_pos={0 0}, upload=39171]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Aug 31 10:37:12.152297525 (44h28m47.505082633s) [upload=39171]: Done uploading segments...
Aug 31 10:37:12.152314989 (44h28m47.505100086s) [upload=39171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 499.028204ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
1.98084529s
FAIL (2025-08-31 10:44:28 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 1.980871469s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
3 out of 15 is failed (threshold=6) [...X.X.X.......] duration:
153.284639ms remote:
"tcp:54.243.86.28:443" response headers:
Cache-Control=private Vary=Accept-Encoding Set-Cookie=_cfuvid=nqFRMjlw7bztCfULNI_Lt3cBumb4nHNMxH8OiqhB06E-1756636739574-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Date=Sun, 31 Aug 2025 10:38:59 GMT Content-Type=text/html Alt-Svc=h3=":443"; ma=86400 X-Cluster-Name=us-east-1-prod-hosting-red Content-Length=166 Location=https://static.storj.io/dcs-satellites Cf-Ray=977be6c609fddddf-IAD Cf-Cache-Status=BYPASS target:
https://www.storj.io/dcs-satellites
FAIL (2025-08-31 10:44:27 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 153.310538ms
core
website
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
45.36063ms target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-31 10:44:27 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp: lookup uptime-checks.gateway.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:29236->100.100.100.100:53: i/o timeout
Duration: 45.378265ms
edge
gateway
vhost
bgp
us1
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
3 out of 60 is failed (threshold=3) [............................................XXX.............] duration:
34.659782ms
FAIL (2025-08-31 10:44:27 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout
Duration: 34.671785ms
edge
authservice
http
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
256.398241ms target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-08-31 10:44:25 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": dial tcp: lookup uptime-checks.gateway.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:29236->100.100.100.100:53: i/o timeout
Duration: 256.476969ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [..........................................X.................] duration:
121.455775ms target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt
FAIL (2025-08-31 10:44:21 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp: lookup link.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:51715->100.100.100.100:53: i/o timeout
Duration: 121.480872ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
687.607623ms target:
https://ap1.storj.io/login
FAIL (2025-08-31 10:44:19 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.157.232:443: i/o timeout
Duration: 687.623753ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........................................XXXXXX............] duration:
594.811314ms
FAIL (2025-08-31 10:44:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Aug 31 10:35:45.949357429 (44h27m21.302142558s) [upload=39166]: Starting upload
Aug 31 10:35:45.949453992 (44h27m21.302239119s) [upload=39166]: Done starting upload
Aug 31 10:35:45.949561915 (44h27m21.302347043s) [upload=39166]: Uploading segments...
Aug 31 10:35:45.949839758 (44h27m21.302624886s) [upload=39166]: Got next segment. Inline: false
Aug 31 10:35:45.949875034 (44h27m21.302660163s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 10:35:45.949880324 (44h27m21.302665453s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 10:35:45.974500057 (44h27m21.327285184s) [seg_pos={0 0}, upload=39166]: Begin upload segment...
Aug 31 10:35:45.974534781 (44h27m21.32731991s) [seg_pos={0 0}, upload=39166]: Done begin upload segment.
Aug 31 10:35:45.974561612 (44h27m21.32734674s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.974622567 (44h27m21.327407705s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975048299 (44h27m21.327833426s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975060932 (44h27m21.327846061s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975453572 (44h27m21.3282387s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975479350 (44h27m21.328264488s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975834078 (44h27m21.328619207s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975889633 (44h27m21.328674761s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976450218 (44h27m21.329235346s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976462792 (44h27m21.32924792s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976837658 (44h27m21.329622786s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976933107 (44h27m21.329718236s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977198717 (44h27m21.329983845s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977335244 (44h27m21.330120372s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977693609 (44h27m21.330478747s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.978139749 (44h27m21.330924877s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:46.010680273 (44h27m21.363465391s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 5.161.89.252:33308->77.90.31.17:4770: write: no route to host; manager closed: conn closed: op
Aug 31 10:35:46.094242937 (44h27m21.447028065s) [upload=39166]: Next returned nil segment
Aug 31 10:35:46.094257233 (44h27m21.447042362s) [upload=39166]: Waiting for error group managing segments...
Aug 31 10:35:46.107643376 (44h27m21.460428494s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.108127397 (44h27m21.460912515s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.109366731 (44h27m21.462151858s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.158408011 (44h27m21.51119313s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.160289913 (44h27m21.513075032s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.175403087 (44h27m21.528188235s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.180837789 (44h27m21.533622927s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.191657048 (44h27m21.544442177s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.201182052 (44h27m21.55396718s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.203261847 (44h27m21.556046995s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:36:15.930363208 (44h27m51.283148347s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.930627486 (44h27m51.283412614s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op
Aug 31 10:36:15.930934504 (44h27m51.283719632s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 5.161.89.252:18584->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931120965 (44h27m51.283906103s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931311063 (44h27m51.284096192s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 5.161.89.252:48956->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931413836 (44h27m51.284198964s) [seg_pos={0 0}, upload=39166]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Aug 31 10:36:15.954490543 (44h27m51.307275682s) [upload=39166]: Done uploading segments...
Aug 31 10:36:15.954518606 (44h27m51.307303744s) [upload=39166]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 594.842182ms
core
uplink
us-select-1
|
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing)antiblip:
4 out of 60 is failed (threshold=3) [..........................................XXXX..............] duration:
8.145866ms target:
http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt
FAIL (2025-08-31 10:44:08 UTC)
orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout
Duration: 8.164973ms
edge
linksharing
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
4 out of 60 is failed (threshold=3) [.......................................XXXX.................] duration:
182.318281ms
FAIL (2025-08-31 10:44:04 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 182.341093ms
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)antiblip:
2 out of 15 is failed (threshold=6) [..........X..X.] duration:
211.562269ms remote:
"tcp:54.243.86.28:443" response headers:
Cf-Ray=977becf37ad21fd4-IAD Last-Modified=Tue, 26 Aug 2025 19:05:15 GMT Set-Cookie=_cfuvid=PpL0dejU4KGmZNWepK7RvPtHJt3a2kIS2JTMXdJ9TNI-1756636992675-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Alt-Svc=h3=":443"; ma=86400 Age=401877 Cf-Cache-Status=HIT Surrogate-Key=www.storj.io 65d78d3b8dbb1f54a94020c7 pageId:67eb0cf63d37c62ac04e0ff0 X-Cluster-Name=us-east-1-prod-hosting-red Surrogate-Control=max-age=2147483647 X-Lambda-Id=c6207948-cc73-40b2-9a2a-e948767f3d5e Vary=Accept-Encoding Date=Sun, 31 Aug 2025 10:43:12 GMT Content-Type=text/html target:
https://www.storj.io/
FAIL (2025-08-31 10:44:00 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 211.585593ms
core
website
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777"} failure (antiblip window 3/60): (authservice.ap1)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
783.495322ms
FAIL (2025-08-31 10:43:58 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp: lookup auth.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:62187->100.100.100.100:53: i/o timeout
Duration: 783.558981ms
edge
authservice
http
bgp
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
1.952251325s
FAIL (2025-08-31 10:43:57 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 1.952270994s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
4.686624484s
FAIL (2025-08-31 10:43:56 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.686645595s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
229.940189ms
FAIL (2025-08-31 10:43:54 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 230.03037ms
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...........................................XXXXXX...........] duration:
545.284571ms
FAIL (2025-08-31 10:43:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Aug 31 10:35:45.949357429 (44h27m21.302142558s) [upload=39166]: Starting upload
Aug 31 10:35:45.949453992 (44h27m21.302239119s) [upload=39166]: Done starting upload
Aug 31 10:35:45.949561915 (44h27m21.302347043s) [upload=39166]: Uploading segments...
Aug 31 10:35:45.949839758 (44h27m21.302624886s) [upload=39166]: Got next segment. Inline: false
Aug 31 10:35:45.949875034 (44h27m21.302660163s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 10:35:45.949880324 (44h27m21.302665453s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 10:35:45.974500057 (44h27m21.327285184s) [seg_pos={0 0}, upload=39166]: Begin upload segment...
Aug 31 10:35:45.974534781 (44h27m21.32731991s) [seg_pos={0 0}, upload=39166]: Done begin upload segment.
Aug 31 10:35:45.974561612 (44h27m21.32734674s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.974622567 (44h27m21.327407705s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975048299 (44h27m21.327833426s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975060932 (44h27m21.327846061s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975453572 (44h27m21.3282387s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975479350 (44h27m21.328264488s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975834078 (44h27m21.328619207s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975889633 (44h27m21.328674761s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976450218 (44h27m21.329235346s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976462792 (44h27m21.32924792s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976837658 (44h27m21.329622786s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976933107 (44h27m21.329718236s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977198717 (44h27m21.329983845s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977335244 (44h27m21.330120372s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977693609 (44h27m21.330478747s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.978139749 (44h27m21.330924877s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:46.010680273 (44h27m21.363465391s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 5.161.89.252:33308->77.90.31.17:4770: write: no route to host; manager closed: conn closed: op
Aug 31 10:35:46.094242937 (44h27m21.447028065s) [upload=39166]: Next returned nil segment
Aug 31 10:35:46.094257233 (44h27m21.447042362s) [upload=39166]: Waiting for error group managing segments...
Aug 31 10:35:46.107643376 (44h27m21.460428494s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.108127397 (44h27m21.460912515s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.109366731 (44h27m21.462151858s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.158408011 (44h27m21.51119313s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.160289913 (44h27m21.513075032s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.175403087 (44h27m21.528188235s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.180837789 (44h27m21.533622927s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.191657048 (44h27m21.544442177s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.201182052 (44h27m21.55396718s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.203261847 (44h27m21.556046995s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:36:15.930363208 (44h27m51.283148347s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.930627486 (44h27m51.283412614s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op
Aug 31 10:36:15.930934504 (44h27m51.283719632s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 5.161.89.252:18584->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931120965 (44h27m51.283906103s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931311063 (44h27m51.284096192s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 5.161.89.252:48956->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931413836 (44h27m51.284198964s) [seg_pos={0 0}, upload=39166]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Aug 31 10:36:15.954490543 (44h27m51.307275682s) [upload=39166]: Done uploading segments...
Aug 31 10:36:15.954518606 (44h27m51.307303744s) [upload=39166]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 545.306212ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
142.864894ms
FAIL (2025-08-31 10:43:50 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 142.904178ms
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [...........................................XX...............] duration:
276.698951ms target:
https://eu1.storj.io/login
FAIL (2025-08-31 10:43:50 UTC)
orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout
Duration: 276.718949ms
core
satellite
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [...........................................X................] duration:
163.007836ms target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt
FAIL (2025-08-31 10:43:48 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp: lookup link.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:51715->100.100.100.100:53: i/o timeout
Duration: 163.02538ms
edge
linksharing
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [............................................XX.XXX..........] duration:
506.643743ms
FAIL (2025-08-31 10:43:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
g 31 10:36:12.150915169 (44h27m47.503700297s) [upload=39171]: Done starting upload
Aug 31 10:36:12.151058448 (44h27m47.503843586s) [upload=39171]: Uploading segments...
Aug 31 10:36:12.151350147 (44h27m47.504135276s) [upload=39171]: Got next segment. Inline: false
Aug 31 10:36:12.151407877 (44h27m47.504193015s) [upload=39171]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 10:36:12.151414058 (44h27m47.504199186s) [upload=39171]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 10:36:12.181724654 (44h27m47.534509782s) [seg_pos={0 0}, upload=39171]: Begin upload segment...
Aug 31 10:36:12.181759048 (44h27m47.534544176s) [seg_pos={0 0}, upload=39171]: Done begin upload segment.
Aug 31 10:36:12.181787412 (44h27m47.534572551s) [noise=true , address=104.36.82.47:4543 , piece_id=OKISSUYRQP4GGIRDDPYFO6MCHDAFJEWCGHLI3P5T7EHBGOD7CYQA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.181865228 (44h27m47.534650367s) [noise=true , address=45.91.116.12:4239 , piece_id=JIN2FAKX4JF3WVYMMICMIFQDIJT7ZZ4XCVCTDTKB7B3OFDTBY6HQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182335613 (44h27m47.535120752s) [noise=true , address=77.90.31.10:4545 , piece_id=R5HG7NAZ2BFJCIYJOMV4BAD3Q7MAANXZJIFUDHBJ56PMITAB7FJA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182439810 (44h27m47.535224938s) [noise=true , address=104.36.82.11:4130 , piece_id=NEJRZC6MR7DTWS62XZZPJXTZW6GVHK4HIK6BXFD3QI46UGQACR3Q, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182733102 (44h27m47.53551823s) [noise=true , address=45.91.116.7:4034 , piece_id=Z5BA6X346T3H2CITKWTLJDZOUOTVCHTXPUDPO2OKKXGRQF3YUMKQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182828772 (44h27m47.535613901s) [noise=true , address=79.127.205.246:4280 , piece_id=BG3RNCBTBDNRY3MMJUQNKJP2RWH57UODQBKK66KK2WEDXA6FJQWA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183134137 (44h27m47.535919276s) [noise=true , address=104.36.82.48:4589 , piece_id=U4VSTDANJ6ZGA6PTOLZBKGBLD6K73SG4FQT6PUIH3QPWIO5A7YQQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183571981 (44h27m47.53635711s) [noise=true , address=77.90.31.13:4629 , piece_id=746YIFKZ7XTJCLHAV2K2TWYRIJP5PFS47B5D37PFT6MERB62QM6Q, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183905159 (44h27m47.536690288s) [noise=true , address=104.36.82.55:4212 , piece_id=BBTOTIURLHZXSY3NUBOJXHOB4U3HJIFEJM2AUO35DFACNAWBEZJA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184080729 (44h27m47.536865858s) [noise=true , address=45.91.116.13:4176 , piece_id=TTOEVZ6NGPERE27QNANZQVNP2QSCVGITHW5EFOV34EJ2KJ2GTVQA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184383600 (44h27m47.537168737s) [noise=true , address=77.90.31.17:4787 , piece_id=YLZWY3HQRSKHUHBOOVY36TATZ5IN7X2C27IXXCJKZZF4MHSRYH4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184492785 (44h27m47.537277914s) [noise=true , address=77.90.31.14:4678 , piece_id=4XGVSFNNY6MGP5IOHJIBXL3S7JIPYAXKHFIAE6BSX5TAKTCJ4JVQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184744750 (44h27m47.537529878s) [noise=true , address=45.91.116.9:4281 , piece_id=EWWXWQPBEDE724UFU2Z7YJ4E5DUI45OMNLHEFCC4OMG7ZQLWWBBQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184850529 (44h27m47.537635657s) [noise=true , address=104.36.82.56:4264 , piece_id=S67WK5JPS46KPUHBQPLDDRGYJOLN6EEVOE5BFYT74U2IIXZFWXTA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185105669 (44h27m47.537890807s) [noise=true , address=77.90.31.21:4891 , piece_id=IKWWTL4THNTJGQ3CQLADO4KY2KFOXPASHSALNGE7NBUAXHPV3MBA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185259740 (44h27m47.538044868s) [noise=true , address=64.49.13.138:4482 , piece_id=JNYXEQ6PTZYSW5ULU55W5ZFEBEW25KXCDA36K47K4ADKODUZOKYA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185545127 (44h27m47.538330255s) [noise=true , address=104.36.82.23:4458 , piece_id=Y5IELCEOPOLHYYU2WEAZM7ZQMS3W6NR5YS64Y4UYD5MYZZQUJY4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185657418 (44h27m47.538442556s) [noise=true , address=104.36.82.44:4883 , piece_id=455DKBRQXXYON4JM5CQ4SE5WR3WTTH3AJUAB6ZEHEHSE4RE6LP4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.220474138 (44h27m47.573259266s) [noise=true , address=77.90.31.21:4891 , piece_id=IKWWTL4THNTJGQ3CQLADO4KY2KFOXPASHSALNGE7NBUAXHPV3MBA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: write tcp 5.161.89.252:27554->77.90.31.21:4891: write: no route to host; conn closed: op
Aug 31 10:36:12.302528983 (44h27m47.655314111s) [upload=39171]: Next returned nil segment
Aug 31 10:36:12.302541757 (44h27m47.655326885s) [upload=39171]: Waiting for error group managing segments...
Aug 31 10:36:12.316638798 (44h27m47.669423926s) [noise=true , address=104.36.82.56:4264 , piece_id=S67WK5JPS46KPUHBQPLDDRGYJOLN6EEVOE5BFYT74U2IIXZFWXTA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.316773852 (44h27m47.66955899s) [noise=true , address=104.36.82.44:4883 , piece_id=455DKBRQXXYON4JM5CQ4SE5WR3WTTH3AJUAB6ZEHEHSE4RE6LP4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317388379 (44h27m47.670173506s) [noise=true , address=104.36.82.47:4543 , piece_id=OKISSUYRQP4GGIRDDPYFO6MCHDAFJEWCGHLI3P5T7EHBGOD7CYQA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317831102 (44h27m47.67061623s) [noise=true , address=104.36.82.23:4458 , piece_id=Y5IELCEOPOLHYYU2WEAZM7ZQMS3W6NR5YS64Y4UYD5MYZZQUJY4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317981115 (44h27m47.670766243s) [noise=true , address=104.36.82.55:4212 , piece_id=BBTOTIURLHZXSY3NUBOJXHOB4U3HJIFEJM2AUO35DFACNAWBEZJA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.318644403 (44h27m47.671429531s) [noise=true , address=104.36.82.11:4130 , piece_id=NEJRZC6MR7DTWS62XZZPJXTZW6GVHK4HIK6BXFD3QI46UGQACR3Q, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.319463665 (44h27m47.672248794s) [noise=true , address=104.36.82.48:4589 , piece_id=U4VSTDANJ6ZGA6PTOLZBKGBLD6K73SG4FQT6PUIH3QPWIO5A7YQQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.365155230 (44h27m47.717940358s) [noise=true , address=45.91.116.13:4176 , piece_id=TTOEVZ6NGPERE27QNANZQVNP2QSCVGITHW5EFOV34EJ2KJ2GTVQA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.382100673 (44h27m47.734885801s) [noise=true , address=45.91.116.9:4281 , piece_id=EWWXWQPBEDE724UFU2Z7YJ4E5DUI45OMNLHEFCC4OMG7ZQLWWBBQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.382477341 (44h27m47.73526248s) [noise=true , address=45.91.116.12:4239 , piece_id=JIN2FAKX4JF3WVYMMICMIFQDIJT7ZZ4XCVCTDTKB7B3OFDTBY6HQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.385851305 (44h27m47.738636442s) [noise=true , address=45.91.116.7:4034 , piece_id=Z5BA6X346T3H2CITKWTLJDZOUOTVCHTXPUDPO2OKKXGRQF3YUMKQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.670547674 (44h27m48.023332772s) [noise=true , address=79.127.205.246:4280 , piece_id=BG3RNCBTBDNRY3MMJUQNKJP2RWH57UODQBKK66KK2WEDXA6FJQWA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.734232925 (44h27m48.087018034s) [noise=true , address=64.49.13.138:4482 , piece_id=JNYXEQ6PTZYSW5ULU55W5ZFEBEW25KXCDA36K47K4ADKODUZOKYA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:37:12.136109118 (44h28m47.488894215s) [noise=true , address=77.90.31.17:4787 , piece_id=YLZWY3HQRSKHUHBOOVY36TATZ5IN7X2C27IXXCJKZZF4MHSRYH4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: rpc: dial tcp 77.90.31.17:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136180632 (44h28m47.48896573s) [noise=true , address=77.90.31.10:4545 , piece_id=R5HG7NAZ2BFJCIYJOMV4BAD3Q7MAANXZJIFUDHBJ56PMITAB7FJA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:1ZFFijCUFET1ShEJuav6uAzFhfraCEcr6seuySYN9ipn4wsZGX, address:77.90.31.10:4545): protocol: rpc: dial tcp 77.90.31.10:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136217150 (44h28m47.489002248s) [noise=true , address=77.90.31.13:4629 , piece_id=746YIFKZ7XTJCLHAV2K2TWYRIJP5PFS47B5D37PFT6MERB62QM6Q, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136244112 (44h28m47.48902921s) [noise=true , address=77.90.31.14:4678 , piece_id=4XGVSFNNY6MGP5IOHJIBXL3S7JIPYAXKHFIAE6BSX5TAKTCJ4JVQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136292463 (44h28m47.48907756s) [seg_pos={0 0}, upload=39171]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Aug 31 10:37:12.152297525 (44h28m47.505082633s) [upload=39171]: Done uploading segments...
Aug 31 10:37:12.152314989 (44h28m47.505100086s) [upload=39171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 506.66389ms
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
4 out of 60 is failed (threshold=3) [..........................................XXXX..............] duration:
17.454271ms target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-31 10:43:46 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout
Duration: 17.470662ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................XXX................] duration:
362.077263ms
FAIL (2025-08-31 10:43:46 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 362.095256ms
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
41.984297ms target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-31 10:43:43 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp: lookup uptime-checks.gateway.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:29236->100.100.100.100:53: i/o timeout
Duration: 42.004806ms
edge
gateway
vhost
bgp
us1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777"} failure (antiblip window 3/60): (authservice.ap1)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
553.704854ms
FAIL (2025-08-31 10:43:42 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp: lookup auth.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:62187->100.100.100.100:53: i/o timeout
Duration: 553.731144ms
edge
authservice
http
bgp
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
282.768437ms target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-08-31 10:43:39 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": dial tcp: lookup uptime-checks.gateway.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:29236->100.100.100.100:53: i/o timeout
Duration: 282.800086ms
edge
gateway
vhost
bgp
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
2.000876574s
FAIL (2025-08-31 10:43:38 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.000931749s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................XX..............] duration:
2.799668825s
FAIL (2025-08-31 10:43:37 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.799685557s
core
uplink
eu1
|
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing)antiblip:
4 out of 60 is failed (threshold=3) [...........................................XXXX.............] duration:
5.021575ms target:
http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt
FAIL (2025-08-31 10:43:37 UTC)
orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout
Duration: 5.041031ms
edge
linksharing
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
653.796588ms
FAIL (2025-08-31 10:43:35 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp: lookup auth.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:62187->100.100.100.100:53: i/o timeout
Duration: 653.814542ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.213187721s
FAIL (2025-08-31 10:43:32 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 2.213216555s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
2 out of 60 is failed (threshold=3) [...........................................XX...............] duration:
133.826908ms
FAIL (2025-08-31 10:43:32 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp: lookup auth.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:62187->100.100.100.100:53: i/o timeout
Duration: 133.841444ms
edge
authservice
http
bgp
us1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [............................................XX..............] duration:
270.52002ms target:
https://eu1.storj.io/login
FAIL (2025-08-31 10:43:30 UTC)
orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout
Duration: 270.538113ms
core
satellite
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [............................................XX..............] duration:
906.659995ms
FAIL (2025-08-31 10:43:29 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 906.681666ms
core
uplink
eu1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
4 out of 15 is failed (threshold=6) [X...X.X.X......] duration:
56.368034ms remote:
"tcp:54.243.86.28:443" response headers:
Cache-Control=private Vary=Accept-Encoding Set-Cookie=_cfuvid=nqFRMjlw7bztCfULNI_Lt3cBumb4nHNMxH8OiqhB06E-1756636739574-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Date=Sun, 31 Aug 2025 10:38:59 GMT Content-Type=text/html Alt-Svc=h3=":443"; ma=86400 X-Cluster-Name=us-east-1-prod-hosting-red Content-Length=166 Location=https://static.storj.io/dcs-satellites Cf-Ray=977be6c609fddddf-IAD Cf-Cache-Status=BYPASS target:
https://www.storj.io/dcs-satellites
FAIL (2025-08-31 10:43:28 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 56.403531ms
core
website
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................................XXX...............] duration:
359.894056ms
FAIL (2025-08-31 10:43:28 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 359.913522ms
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
3 out of 60 is failed (threshold=3) [.............................................XXX............] duration:
37.607203ms
FAIL (2025-08-31 10:43:26 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout
Duration: 37.630476ms
edge
authservice
http
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
4 out of 60 is failed (threshold=3) [........................................XXXX................] duration:
183.662393ms
FAIL (2025-08-31 10:43:23 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 183.683733ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................XX.............] duration:
2.825768158s
FAIL (2025-08-31 10:43:20 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.82579556s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
193.625801ms
FAIL (2025-08-31 10:43:19 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 193.650607ms
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [............................................XXXXXX..........] duration:
596.213245ms
FAIL (2025-08-31 10:43:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Aug 31 10:35:45.949357429 (44h27m21.302142558s) [upload=39166]: Starting upload
Aug 31 10:35:45.949453992 (44h27m21.302239119s) [upload=39166]: Done starting upload
Aug 31 10:35:45.949561915 (44h27m21.302347043s) [upload=39166]: Uploading segments...
Aug 31 10:35:45.949839758 (44h27m21.302624886s) [upload=39166]: Got next segment. Inline: false
Aug 31 10:35:45.949875034 (44h27m21.302660163s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 10:35:45.949880324 (44h27m21.302665453s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 10:35:45.974500057 (44h27m21.327285184s) [seg_pos={0 0}, upload=39166]: Begin upload segment...
Aug 31 10:35:45.974534781 (44h27m21.32731991s) [seg_pos={0 0}, upload=39166]: Done begin upload segment.
Aug 31 10:35:45.974561612 (44h27m21.32734674s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.974622567 (44h27m21.327407705s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975048299 (44h27m21.327833426s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975060932 (44h27m21.327846061s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975453572 (44h27m21.3282387s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975479350 (44h27m21.328264488s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975834078 (44h27m21.328619207s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975889633 (44h27m21.328674761s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976450218 (44h27m21.329235346s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976462792 (44h27m21.32924792s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976837658 (44h27m21.329622786s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976933107 (44h27m21.329718236s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977198717 (44h27m21.329983845s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977335244 (44h27m21.330120372s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977693609 (44h27m21.330478747s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.978139749 (44h27m21.330924877s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:46.010680273 (44h27m21.363465391s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 5.161.89.252:33308->77.90.31.17:4770: write: no route to host; manager closed: conn closed: op
Aug 31 10:35:46.094242937 (44h27m21.447028065s) [upload=39166]: Next returned nil segment
Aug 31 10:35:46.094257233 (44h27m21.447042362s) [upload=39166]: Waiting for error group managing segments...
Aug 31 10:35:46.107643376 (44h27m21.460428494s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.108127397 (44h27m21.460912515s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.109366731 (44h27m21.462151858s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.158408011 (44h27m21.51119313s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.160289913 (44h27m21.513075032s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.175403087 (44h27m21.528188235s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.180837789 (44h27m21.533622927s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.191657048 (44h27m21.544442177s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.201182052 (44h27m21.55396718s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.203261847 (44h27m21.556046995s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:36:15.930363208 (44h27m51.283148347s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.930627486 (44h27m51.283412614s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op
Aug 31 10:36:15.930934504 (44h27m51.283719632s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 5.161.89.252:18584->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931120965 (44h27m51.283906103s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931311063 (44h27m51.284096192s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 5.161.89.252:48956->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931413836 (44h27m51.284198964s) [seg_pos={0 0}, upload=39166]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Aug 31 10:36:15.954490543 (44h27m51.307275682s) [upload=39166]: Done uploading segments...
Aug 31 10:36:15.954518606 (44h27m51.307303744s) [upload=39166]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 596.254924ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
4.797472082s
FAIL (2025-08-31 10:43:15 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.797499604s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.ap1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
251.536828ms target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011
FAIL (2025-08-31 10:43:15 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwlk53gfhsby563unjoqoivp56jq&Signature=Q7FkmqtUH%2BgeU7PdNdPIPgOXqhY%3D&Expires=1771549011": dial tcp: lookup uptime-checks.gateway.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:29236->100.100.100.100:53: i/o timeout
Duration: 251.560704ms
edge
gateway
vhost
bgp
ap1
|
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)antiblip:
3 out of 15 is failed (threshold=6) [X..........X..X] duration:
172.764455ms remote:
"tcp:54.243.86.28:443" response headers:
Cf-Ray=977becf37ad21fd4-IAD Last-Modified=Tue, 26 Aug 2025 19:05:15 GMT Set-Cookie=_cfuvid=PpL0dejU4KGmZNWepK7RvPtHJt3a2kIS2JTMXdJ9TNI-1756636992675-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Alt-Svc=h3=":443"; ma=86400 Age=401877 Cf-Cache-Status=HIT Surrogate-Key=www.storj.io 65d78d3b8dbb1f54a94020c7 pageId:67eb0cf63d37c62ac04e0ff0 X-Cluster-Name=us-east-1-prod-hosting-red Surrogate-Control=max-age=2147483647 X-Lambda-Id=c6207948-cc73-40b2-9a2a-e948767f3d5e Vary=Accept-Encoding Date=Sun, 31 Aug 2025 10:43:12 GMT Content-Type=text/html target:
https://www.storj.io/
FAIL (2025-08-31 10:43:12 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 172.778231ms
core
website
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
4 out of 60 is failed (threshold=3) [...........................................XXXX.............] duration:
26.926665ms target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-31 10:43:12 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout
Duration: 26.947624ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
641.122066ms
FAIL (2025-08-31 10:43:11 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp: lookup auth.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:62187->100.100.100.100:53: i/o timeout
Duration: 641.14092ms
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing)antiblip:
4 out of 60 is failed (threshold=3) [............................................XXXX............] duration:
9.038586ms target:
http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt
FAIL (2025-08-31 10:43:09 UTC)
orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout
Duration: 9.107947ms
edge
linksharing
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
2 out of 60 is failed (threshold=3) [............................................XX..............] duration:
145.498199ms
FAIL (2025-08-31 10:43:09 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp: lookup auth.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:62187->100.100.100.100:53: i/o timeout
Duration: 145.531112ms
edge
authservice
http
bgp
us1
|