Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)antiblip:
5 out of 15 is failed (threshold=6) [.......XX..X.XX] duration:
143.51718ms remote:
"tcp:34.202.203.47:443" response headers:
Surrogate-Control=max-age=2147483647 X-Lambda-Id=c6207948-cc73-40b2-9a2a-e948767f3d5e Vary=Accept-Encoding Date=Sun, 31 Aug 2025 11:20:03 GMT Content-Type=text/html Cf-Ray=977c22eebe922894-IAD Cf-Cache-Status=HIT Age=404088 Surrogate-Key=www.storj.io 65d78d3b8dbb1f54a94020c7 pageId:67eb0cf63d37c62ac04e0ff0 Last-Modified=Tue, 26 Aug 2025 19:05:15 GMT Set-Cookie=_cfuvid=Xau6yr7KyruRisw3OyVYeEiVuHK_ZiQNmPRRk6iM4us-1756639203762-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Alt-Svc=h3=":443"; ma=86400 X-Cluster-Name=us-east-1-prod-hosting-red target:
https://www.storj.io/
FAIL (2025-08-31 11:20:03 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 143.552094ms
core
website
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
72.914891ms
FAIL (2025-08-31 11:19:59 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout
Duration: 72.94088ms
edge
authservice
http
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
2 out of 15 is failed (threshold=6) [X.........X....] duration:
72.525033ms remote:
"tcp:54.243.86.28:443" response headers:
Cache-Control=private Alt-Svc=h3=":443"; ma=86400 X-Cluster-Name=us-east-1-prod-hosting-red Date=Sun, 31 Aug 2025 11:17:50 GMT Content-Length=166 Cf-Ray=977c1fb069c3d697-IAD Cf-Cache-Status=BYPASS Vary=Accept-Encoding Set-Cookie=_cfuvid=Dmeli_aGxBKFgSt8S3gcL6iZnzp0F43lPBd5BQ_WGoU-1756639070832-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Content-Type=text/html Location=https://static.storj.io/dcs-satellites target:
https://www.storj.io/dcs-satellites
FAIL (2025-08-31 11:19:45 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 72.575848ms
core
website
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.953668617s
FAIL (2025-08-31 11:19:34 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.953687775s
core
uplink
eu1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
132.083534ms
FAIL (2025-08-31 11:19:30 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp: lookup auth.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:62187->100.100.100.100:53: i/o timeout
Duration: 132.101228ms
edge
authservice
http
bgp
us1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
2 out of 15 is failed (threshold=6) [.X.........X...] duration:
61.004675ms remote:
"tcp:54.243.86.28:443" response headers:
Cache-Control=private Alt-Svc=h3=":443"; ma=86400 X-Cluster-Name=us-east-1-prod-hosting-red Date=Sun, 31 Aug 2025 11:17:50 GMT Content-Length=166 Cf-Ray=977c1fb069c3d697-IAD Cf-Cache-Status=BYPASS Vary=Accept-Encoding Set-Cookie=_cfuvid=Dmeli_aGxBKFgSt8S3gcL6iZnzp0F43lPBd5BQ_WGoU-1756639070832-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Content-Type=text/html Location=https://static.storj.io/dcs-satellites target:
https://www.storj.io/dcs-satellites
FAIL (2025-08-31 11:19:26 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 61.029833ms
core
website
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
863.843696ms
FAIL (2025-08-31 11:19:15 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 863.860888ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.XXX........................................................] duration:
642.168285ms
FAIL (2025-08-31 11:19:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
g 31 10:36:12.150915169 (44h27m47.503700297s) [upload=39171]: Done starting upload
Aug 31 10:36:12.151058448 (44h27m47.503843586s) [upload=39171]: Uploading segments...
Aug 31 10:36:12.151350147 (44h27m47.504135276s) [upload=39171]: Got next segment. Inline: false
Aug 31 10:36:12.151407877 (44h27m47.504193015s) [upload=39171]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 10:36:12.151414058 (44h27m47.504199186s) [upload=39171]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 10:36:12.181724654 (44h27m47.534509782s) [seg_pos={0 0}, upload=39171]: Begin upload segment...
Aug 31 10:36:12.181759048 (44h27m47.534544176s) [seg_pos={0 0}, upload=39171]: Done begin upload segment.
Aug 31 10:36:12.181787412 (44h27m47.534572551s) [noise=true , address=104.36.82.47:4543 , piece_id=OKISSUYRQP4GGIRDDPYFO6MCHDAFJEWCGHLI3P5T7EHBGOD7CYQA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.181865228 (44h27m47.534650367s) [noise=true , address=45.91.116.12:4239 , piece_id=JIN2FAKX4JF3WVYMMICMIFQDIJT7ZZ4XCVCTDTKB7B3OFDTBY6HQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182335613 (44h27m47.535120752s) [noise=true , address=77.90.31.10:4545 , piece_id=R5HG7NAZ2BFJCIYJOMV4BAD3Q7MAANXZJIFUDHBJ56PMITAB7FJA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182439810 (44h27m47.535224938s) [noise=true , address=104.36.82.11:4130 , piece_id=NEJRZC6MR7DTWS62XZZPJXTZW6GVHK4HIK6BXFD3QI46UGQACR3Q, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182733102 (44h27m47.53551823s) [noise=true , address=45.91.116.7:4034 , piece_id=Z5BA6X346T3H2CITKWTLJDZOUOTVCHTXPUDPO2OKKXGRQF3YUMKQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182828772 (44h27m47.535613901s) [noise=true , address=79.127.205.246:4280 , piece_id=BG3RNCBTBDNRY3MMJUQNKJP2RWH57UODQBKK66KK2WEDXA6FJQWA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183134137 (44h27m47.535919276s) [noise=true , address=104.36.82.48:4589 , piece_id=U4VSTDANJ6ZGA6PTOLZBKGBLD6K73SG4FQT6PUIH3QPWIO5A7YQQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183571981 (44h27m47.53635711s) [noise=true , address=77.90.31.13:4629 , piece_id=746YIFKZ7XTJCLHAV2K2TWYRIJP5PFS47B5D37PFT6MERB62QM6Q, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183905159 (44h27m47.536690288s) [noise=true , address=104.36.82.55:4212 , piece_id=BBTOTIURLHZXSY3NUBOJXHOB4U3HJIFEJM2AUO35DFACNAWBEZJA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184080729 (44h27m47.536865858s) [noise=true , address=45.91.116.13:4176 , piece_id=TTOEVZ6NGPERE27QNANZQVNP2QSCVGITHW5EFOV34EJ2KJ2GTVQA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184383600 (44h27m47.537168737s) [noise=true , address=77.90.31.17:4787 , piece_id=YLZWY3HQRSKHUHBOOVY36TATZ5IN7X2C27IXXCJKZZF4MHSRYH4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184492785 (44h27m47.537277914s) [noise=true , address=77.90.31.14:4678 , piece_id=4XGVSFNNY6MGP5IOHJIBXL3S7JIPYAXKHFIAE6BSX5TAKTCJ4JVQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184744750 (44h27m47.537529878s) [noise=true , address=45.91.116.9:4281 , piece_id=EWWXWQPBEDE724UFU2Z7YJ4E5DUI45OMNLHEFCC4OMG7ZQLWWBBQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184850529 (44h27m47.537635657s) [noise=true , address=104.36.82.56:4264 , piece_id=S67WK5JPS46KPUHBQPLDDRGYJOLN6EEVOE5BFYT74U2IIXZFWXTA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185105669 (44h27m47.537890807s) [noise=true , address=77.90.31.21:4891 , piece_id=IKWWTL4THNTJGQ3CQLADO4KY2KFOXPASHSALNGE7NBUAXHPV3MBA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185259740 (44h27m47.538044868s) [noise=true , address=64.49.13.138:4482 , piece_id=JNYXEQ6PTZYSW5ULU55W5ZFEBEW25KXCDA36K47K4ADKODUZOKYA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185545127 (44h27m47.538330255s) [noise=true , address=104.36.82.23:4458 , piece_id=Y5IELCEOPOLHYYU2WEAZM7ZQMS3W6NR5YS64Y4UYD5MYZZQUJY4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185657418 (44h27m47.538442556s) [noise=true , address=104.36.82.44:4883 , piece_id=455DKBRQXXYON4JM5CQ4SE5WR3WTTH3AJUAB6ZEHEHSE4RE6LP4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.220474138 (44h27m47.573259266s) [noise=true , address=77.90.31.21:4891 , piece_id=IKWWTL4THNTJGQ3CQLADO4KY2KFOXPASHSALNGE7NBUAXHPV3MBA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: write tcp 5.161.89.252:27554->77.90.31.21:4891: write: no route to host; conn closed: op
Aug 31 10:36:12.302528983 (44h27m47.655314111s) [upload=39171]: Next returned nil segment
Aug 31 10:36:12.302541757 (44h27m47.655326885s) [upload=39171]: Waiting for error group managing segments...
Aug 31 10:36:12.316638798 (44h27m47.669423926s) [noise=true , address=104.36.82.56:4264 , piece_id=S67WK5JPS46KPUHBQPLDDRGYJOLN6EEVOE5BFYT74U2IIXZFWXTA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.316773852 (44h27m47.66955899s) [noise=true , address=104.36.82.44:4883 , piece_id=455DKBRQXXYON4JM5CQ4SE5WR3WTTH3AJUAB6ZEHEHSE4RE6LP4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317388379 (44h27m47.670173506s) [noise=true , address=104.36.82.47:4543 , piece_id=OKISSUYRQP4GGIRDDPYFO6MCHDAFJEWCGHLI3P5T7EHBGOD7CYQA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317831102 (44h27m47.67061623s) [noise=true , address=104.36.82.23:4458 , piece_id=Y5IELCEOPOLHYYU2WEAZM7ZQMS3W6NR5YS64Y4UYD5MYZZQUJY4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317981115 (44h27m47.670766243s) [noise=true , address=104.36.82.55:4212 , piece_id=BBTOTIURLHZXSY3NUBOJXHOB4U3HJIFEJM2AUO35DFACNAWBEZJA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.318644403 (44h27m47.671429531s) [noise=true , address=104.36.82.11:4130 , piece_id=NEJRZC6MR7DTWS62XZZPJXTZW6GVHK4HIK6BXFD3QI46UGQACR3Q, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.319463665 (44h27m47.672248794s) [noise=true , address=104.36.82.48:4589 , piece_id=U4VSTDANJ6ZGA6PTOLZBKGBLD6K73SG4FQT6PUIH3QPWIO5A7YQQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.365155230 (44h27m47.717940358s) [noise=true , address=45.91.116.13:4176 , piece_id=TTOEVZ6NGPERE27QNANZQVNP2QSCVGITHW5EFOV34EJ2KJ2GTVQA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.382100673 (44h27m47.734885801s) [noise=true , address=45.91.116.9:4281 , piece_id=EWWXWQPBEDE724UFU2Z7YJ4E5DUI45OMNLHEFCC4OMG7ZQLWWBBQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.382477341 (44h27m47.73526248s) [noise=true , address=45.91.116.12:4239 , piece_id=JIN2FAKX4JF3WVYMMICMIFQDIJT7ZZ4XCVCTDTKB7B3OFDTBY6HQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.385851305 (44h27m47.738636442s) [noise=true , address=45.91.116.7:4034 , piece_id=Z5BA6X346T3H2CITKWTLJDZOUOTVCHTXPUDPO2OKKXGRQF3YUMKQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.670547674 (44h27m48.023332772s) [noise=true , address=79.127.205.246:4280 , piece_id=BG3RNCBTBDNRY3MMJUQNKJP2RWH57UODQBKK66KK2WEDXA6FJQWA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.734232925 (44h27m48.087018034s) [noise=true , address=64.49.13.138:4482 , piece_id=JNYXEQ6PTZYSW5ULU55W5ZFEBEW25KXCDA36K47K4ADKODUZOKYA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:37:12.136109118 (44h28m47.488894215s) [noise=true , address=77.90.31.17:4787 , piece_id=YLZWY3HQRSKHUHBOOVY36TATZ5IN7X2C27IXXCJKZZF4MHSRYH4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: rpc: dial tcp 77.90.31.17:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136180632 (44h28m47.48896573s) [noise=true , address=77.90.31.10:4545 , piece_id=R5HG7NAZ2BFJCIYJOMV4BAD3Q7MAANXZJIFUDHBJ56PMITAB7FJA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:1ZFFijCUFET1ShEJuav6uAzFhfraCEcr6seuySYN9ipn4wsZGX, address:77.90.31.10:4545): protocol: rpc: dial tcp 77.90.31.10:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136217150 (44h28m47.489002248s) [noise=true , address=77.90.31.13:4629 , piece_id=746YIFKZ7XTJCLHAV2K2TWYRIJP5PFS47B5D37PFT6MERB62QM6Q, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136244112 (44h28m47.48902921s) [noise=true , address=77.90.31.14:4678 , piece_id=4XGVSFNNY6MGP5IOHJIBXL3S7JIPYAXKHFIAE6BSX5TAKTCJ4JVQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136292463 (44h28m47.48907756s) [seg_pos={0 0}, upload=39171]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Aug 31 10:37:12.152297525 (44h28m47.505082633s) [upload=39171]: Done uploading segments...
Aug 31 10:37:12.152314989 (44h28m47.505100086s) [upload=39171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 642.198603ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
2.872931213s
FAIL (2025-08-31 11:19:10 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.87295032s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
514.980077ms
FAIL (2025-08-31 11:19:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Aug 31 10:35:45.949357429 (44h27m21.302142558s) [upload=39166]: Starting upload
Aug 31 10:35:45.949453992 (44h27m21.302239119s) [upload=39166]: Done starting upload
Aug 31 10:35:45.949561915 (44h27m21.302347043s) [upload=39166]: Uploading segments...
Aug 31 10:35:45.949839758 (44h27m21.302624886s) [upload=39166]: Got next segment. Inline: false
Aug 31 10:35:45.949875034 (44h27m21.302660163s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 10:35:45.949880324 (44h27m21.302665453s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 10:35:45.974500057 (44h27m21.327285184s) [seg_pos={0 0}, upload=39166]: Begin upload segment...
Aug 31 10:35:45.974534781 (44h27m21.32731991s) [seg_pos={0 0}, upload=39166]: Done begin upload segment.
Aug 31 10:35:45.974561612 (44h27m21.32734674s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.974622567 (44h27m21.327407705s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975048299 (44h27m21.327833426s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975060932 (44h27m21.327846061s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975453572 (44h27m21.3282387s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975479350 (44h27m21.328264488s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975834078 (44h27m21.328619207s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975889633 (44h27m21.328674761s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976450218 (44h27m21.329235346s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976462792 (44h27m21.32924792s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976837658 (44h27m21.329622786s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976933107 (44h27m21.329718236s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977198717 (44h27m21.329983845s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977335244 (44h27m21.330120372s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977693609 (44h27m21.330478747s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.978139749 (44h27m21.330924877s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:46.010680273 (44h27m21.363465391s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 5.161.89.252:33308->77.90.31.17:4770: write: no route to host; manager closed: conn closed: op
Aug 31 10:35:46.094242937 (44h27m21.447028065s) [upload=39166]: Next returned nil segment
Aug 31 10:35:46.094257233 (44h27m21.447042362s) [upload=39166]: Waiting for error group managing segments...
Aug 31 10:35:46.107643376 (44h27m21.460428494s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.108127397 (44h27m21.460912515s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.109366731 (44h27m21.462151858s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.158408011 (44h27m21.51119313s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.160289913 (44h27m21.513075032s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.175403087 (44h27m21.528188235s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.180837789 (44h27m21.533622927s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.191657048 (44h27m21.544442177s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.201182052 (44h27m21.55396718s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.203261847 (44h27m21.556046995s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:36:15.930363208 (44h27m51.283148347s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.930627486 (44h27m51.283412614s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op
Aug 31 10:36:15.930934504 (44h27m51.283719632s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 5.161.89.252:18584->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931120965 (44h27m51.283906103s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931311063 (44h27m51.284096192s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 5.161.89.252:48956->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931413836 (44h27m51.284198964s) [seg_pos={0 0}, upload=39166]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Aug 31 10:36:15.954490543 (44h27m51.307275682s) [upload=39166]: Done uploading segments...
Aug 31 10:36:15.954518606 (44h27m51.307303744s) [upload=39166]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 515.00287ms
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
2 out of 15 is failed (threshold=6) [..X.........X..] duration:
56.873354ms remote:
"tcp:54.243.86.28:443" response headers:
Cache-Control=private Alt-Svc=h3=":443"; ma=86400 X-Cluster-Name=us-east-1-prod-hosting-red Date=Sun, 31 Aug 2025 11:17:50 GMT Content-Length=166 Cf-Ray=977c1fb069c3d697-IAD Cf-Cache-Status=BYPASS Vary=Accept-Encoding Set-Cookie=_cfuvid=Dmeli_aGxBKFgSt8S3gcL6iZnzp0F43lPBd5BQ_WGoU-1756639070832-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Content-Type=text/html Location=https://static.storj.io/dcs-satellites target:
https://www.storj.io/dcs-satellites
FAIL (2025-08-31 11:19:05 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 56.904242ms
core
website
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
166.252093ms
FAIL (2025-08-31 11:19:04 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp: lookup auth.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:62187->100.100.100.100:53: i/o timeout
Duration: 166.284353ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
168.935306ms
FAIL (2025-08-31 11:19:02 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 168.957798ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
4.798054487s
FAIL (2025-08-31 11:19:02 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.798080125s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)antiblip:
5 out of 15 is failed (threshold=6) [X.......XX..X.X] duration:
79.794467ms remote:
"tcp:54.243.86.28:443" response headers:
Vary=Accept-Encoding X-Cluster-Name=us-east-1-prod-hosting-red Content-Type=text/html Set-Cookie=_cfuvid=JmBQiiN6jJMz_LUHir9xRR4JN.fDFN67kRJGrDk6wNo-1756639132181-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Date=Sun, 31 Aug 2025 11:18:52 GMT Cf-Ray=977c212fcee5c98f-IAD Age=404017 Surrogate-Control=max-age=2147483647 X-Lambda-Id=c6207948-cc73-40b2-9a2a-e948767f3d5e Alt-Svc=h3=":443"; ma=86400 Cf-Cache-Status=HIT Last-Modified=Tue, 26 Aug 2025 19:05:15 GMT Surrogate-Key=www.storj.io 65d78d3b8dbb1f54a94020c7 pageId:67eb0cf63d37c62ac04e0ff0 target:
https://www.storj.io/
FAIL (2025-08-31 11:18:52 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 79.821859ms
core
website
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
3 out of 60 is failed (threshold=3) [XXX.........................................................] duration:
34.96986ms
FAIL (2025-08-31 11:18:46 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout
Duration: 35.016778ms
edge
authservice
http
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
141.597938ms
FAIL (2025-08-31 11:18:42 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp: lookup auth.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:62187->100.100.100.100:53: i/o timeout
Duration: 141.651019ms
edge
authservice
http
bgp
us1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
2 out of 15 is failed (threshold=6) [...X.........X.] duration:
185.921837ms remote:
"tcp:54.243.86.28:443" response headers:
Cache-Control=private Alt-Svc=h3=":443"; ma=86400 X-Cluster-Name=us-east-1-prod-hosting-red Date=Sun, 31 Aug 2025 11:17:50 GMT Content-Length=166 Cf-Ray=977c1fb069c3d697-IAD Cf-Cache-Status=BYPASS Vary=Accept-Encoding Set-Cookie=_cfuvid=Dmeli_aGxBKFgSt8S3gcL6iZnzp0F43lPBd5BQ_WGoU-1756639070832-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Content-Type=text/html Location=https://static.storj.io/dcs-satellites target:
https://www.storj.io/dcs-satellites
FAIL (2025-08-31 11:18:30 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 185.944809ms
core
website
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
2 out of 60 is failed (threshold=3) [..XX........................................................] duration:
142.795042ms
FAIL (2025-08-31 11:18:26 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp: lookup auth.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:62187->100.100.100.100:53: i/o timeout
Duration: 142.815622ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X.XXX.......................................................] duration:
513.701139ms
FAIL (2025-08-31 11:18:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
g 31 10:36:12.150915169 (44h27m47.503700297s) [upload=39171]: Done starting upload
Aug 31 10:36:12.151058448 (44h27m47.503843586s) [upload=39171]: Uploading segments...
Aug 31 10:36:12.151350147 (44h27m47.504135276s) [upload=39171]: Got next segment. Inline: false
Aug 31 10:36:12.151407877 (44h27m47.504193015s) [upload=39171]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 10:36:12.151414058 (44h27m47.504199186s) [upload=39171]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 10:36:12.181724654 (44h27m47.534509782s) [seg_pos={0 0}, upload=39171]: Begin upload segment...
Aug 31 10:36:12.181759048 (44h27m47.534544176s) [seg_pos={0 0}, upload=39171]: Done begin upload segment.
Aug 31 10:36:12.181787412 (44h27m47.534572551s) [noise=true , address=104.36.82.47:4543 , piece_id=OKISSUYRQP4GGIRDDPYFO6MCHDAFJEWCGHLI3P5T7EHBGOD7CYQA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.181865228 (44h27m47.534650367s) [noise=true , address=45.91.116.12:4239 , piece_id=JIN2FAKX4JF3WVYMMICMIFQDIJT7ZZ4XCVCTDTKB7B3OFDTBY6HQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182335613 (44h27m47.535120752s) [noise=true , address=77.90.31.10:4545 , piece_id=R5HG7NAZ2BFJCIYJOMV4BAD3Q7MAANXZJIFUDHBJ56PMITAB7FJA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182439810 (44h27m47.535224938s) [noise=true , address=104.36.82.11:4130 , piece_id=NEJRZC6MR7DTWS62XZZPJXTZW6GVHK4HIK6BXFD3QI46UGQACR3Q, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182733102 (44h27m47.53551823s) [noise=true , address=45.91.116.7:4034 , piece_id=Z5BA6X346T3H2CITKWTLJDZOUOTVCHTXPUDPO2OKKXGRQF3YUMKQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182828772 (44h27m47.535613901s) [noise=true , address=79.127.205.246:4280 , piece_id=BG3RNCBTBDNRY3MMJUQNKJP2RWH57UODQBKK66KK2WEDXA6FJQWA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183134137 (44h27m47.535919276s) [noise=true , address=104.36.82.48:4589 , piece_id=U4VSTDANJ6ZGA6PTOLZBKGBLD6K73SG4FQT6PUIH3QPWIO5A7YQQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183571981 (44h27m47.53635711s) [noise=true , address=77.90.31.13:4629 , piece_id=746YIFKZ7XTJCLHAV2K2TWYRIJP5PFS47B5D37PFT6MERB62QM6Q, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183905159 (44h27m47.536690288s) [noise=true , address=104.36.82.55:4212 , piece_id=BBTOTIURLHZXSY3NUBOJXHOB4U3HJIFEJM2AUO35DFACNAWBEZJA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184080729 (44h27m47.536865858s) [noise=true , address=45.91.116.13:4176 , piece_id=TTOEVZ6NGPERE27QNANZQVNP2QSCVGITHW5EFOV34EJ2KJ2GTVQA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184383600 (44h27m47.537168737s) [noise=true , address=77.90.31.17:4787 , piece_id=YLZWY3HQRSKHUHBOOVY36TATZ5IN7X2C27IXXCJKZZF4MHSRYH4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184492785 (44h27m47.537277914s) [noise=true , address=77.90.31.14:4678 , piece_id=4XGVSFNNY6MGP5IOHJIBXL3S7JIPYAXKHFIAE6BSX5TAKTCJ4JVQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184744750 (44h27m47.537529878s) [noise=true , address=45.91.116.9:4281 , piece_id=EWWXWQPBEDE724UFU2Z7YJ4E5DUI45OMNLHEFCC4OMG7ZQLWWBBQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184850529 (44h27m47.537635657s) [noise=true , address=104.36.82.56:4264 , piece_id=S67WK5JPS46KPUHBQPLDDRGYJOLN6EEVOE5BFYT74U2IIXZFWXTA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185105669 (44h27m47.537890807s) [noise=true , address=77.90.31.21:4891 , piece_id=IKWWTL4THNTJGQ3CQLADO4KY2KFOXPASHSALNGE7NBUAXHPV3MBA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185259740 (44h27m47.538044868s) [noise=true , address=64.49.13.138:4482 , piece_id=JNYXEQ6PTZYSW5ULU55W5ZFEBEW25KXCDA36K47K4ADKODUZOKYA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185545127 (44h27m47.538330255s) [noise=true , address=104.36.82.23:4458 , piece_id=Y5IELCEOPOLHYYU2WEAZM7ZQMS3W6NR5YS64Y4UYD5MYZZQUJY4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185657418 (44h27m47.538442556s) [noise=true , address=104.36.82.44:4883 , piece_id=455DKBRQXXYON4JM5CQ4SE5WR3WTTH3AJUAB6ZEHEHSE4RE6LP4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.220474138 (44h27m47.573259266s) [noise=true , address=77.90.31.21:4891 , piece_id=IKWWTL4THNTJGQ3CQLADO4KY2KFOXPASHSALNGE7NBUAXHPV3MBA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: write tcp 5.161.89.252:27554->77.90.31.21:4891: write: no route to host; conn closed: op
Aug 31 10:36:12.302528983 (44h27m47.655314111s) [upload=39171]: Next returned nil segment
Aug 31 10:36:12.302541757 (44h27m47.655326885s) [upload=39171]: Waiting for error group managing segments...
Aug 31 10:36:12.316638798 (44h27m47.669423926s) [noise=true , address=104.36.82.56:4264 , piece_id=S67WK5JPS46KPUHBQPLDDRGYJOLN6EEVOE5BFYT74U2IIXZFWXTA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.316773852 (44h27m47.66955899s) [noise=true , address=104.36.82.44:4883 , piece_id=455DKBRQXXYON4JM5CQ4SE5WR3WTTH3AJUAB6ZEHEHSE4RE6LP4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317388379 (44h27m47.670173506s) [noise=true , address=104.36.82.47:4543 , piece_id=OKISSUYRQP4GGIRDDPYFO6MCHDAFJEWCGHLI3P5T7EHBGOD7CYQA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317831102 (44h27m47.67061623s) [noise=true , address=104.36.82.23:4458 , piece_id=Y5IELCEOPOLHYYU2WEAZM7ZQMS3W6NR5YS64Y4UYD5MYZZQUJY4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317981115 (44h27m47.670766243s) [noise=true , address=104.36.82.55:4212 , piece_id=BBTOTIURLHZXSY3NUBOJXHOB4U3HJIFEJM2AUO35DFACNAWBEZJA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.318644403 (44h27m47.671429531s) [noise=true , address=104.36.82.11:4130 , piece_id=NEJRZC6MR7DTWS62XZZPJXTZW6GVHK4HIK6BXFD3QI46UGQACR3Q, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.319463665 (44h27m47.672248794s) [noise=true , address=104.36.82.48:4589 , piece_id=U4VSTDANJ6ZGA6PTOLZBKGBLD6K73SG4FQT6PUIH3QPWIO5A7YQQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.365155230 (44h27m47.717940358s) [noise=true , address=45.91.116.13:4176 , piece_id=TTOEVZ6NGPERE27QNANZQVNP2QSCVGITHW5EFOV34EJ2KJ2GTVQA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.382100673 (44h27m47.734885801s) [noise=true , address=45.91.116.9:4281 , piece_id=EWWXWQPBEDE724UFU2Z7YJ4E5DUI45OMNLHEFCC4OMG7ZQLWWBBQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.382477341 (44h27m47.73526248s) [noise=true , address=45.91.116.12:4239 , piece_id=JIN2FAKX4JF3WVYMMICMIFQDIJT7ZZ4XCVCTDTKB7B3OFDTBY6HQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.385851305 (44h27m47.738636442s) [noise=true , address=45.91.116.7:4034 , piece_id=Z5BA6X346T3H2CITKWTLJDZOUOTVCHTXPUDPO2OKKXGRQF3YUMKQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.670547674 (44h27m48.023332772s) [noise=true , address=79.127.205.246:4280 , piece_id=BG3RNCBTBDNRY3MMJUQNKJP2RWH57UODQBKK66KK2WEDXA6FJQWA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.734232925 (44h27m48.087018034s) [noise=true , address=64.49.13.138:4482 , piece_id=JNYXEQ6PTZYSW5ULU55W5ZFEBEW25KXCDA36K47K4ADKODUZOKYA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:37:12.136109118 (44h28m47.488894215s) [noise=true , address=77.90.31.17:4787 , piece_id=YLZWY3HQRSKHUHBOOVY36TATZ5IN7X2C27IXXCJKZZF4MHSRYH4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: rpc: dial tcp 77.90.31.17:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136180632 (44h28m47.48896573s) [noise=true , address=77.90.31.10:4545 , piece_id=R5HG7NAZ2BFJCIYJOMV4BAD3Q7MAANXZJIFUDHBJ56PMITAB7FJA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:1ZFFijCUFET1ShEJuav6uAzFhfraCEcr6seuySYN9ipn4wsZGX, address:77.90.31.10:4545): protocol: rpc: dial tcp 77.90.31.10:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136217150 (44h28m47.489002248s) [noise=true , address=77.90.31.13:4629 , piece_id=746YIFKZ7XTJCLHAV2K2TWYRIJP5PFS47B5D37PFT6MERB62QM6Q, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136244112 (44h28m47.48902921s) [noise=true , address=77.90.31.14:4678 , piece_id=4XGVSFNNY6MGP5IOHJIBXL3S7JIPYAXKHFIAE6BSX5TAKTCJ4JVQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136292463 (44h28m47.48907756s) [seg_pos={0 0}, upload=39171]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Aug 31 10:37:12.152297525 (44h28m47.505082633s) [upload=39171]: Done uploading segments...
Aug 31 10:37:12.152314989 (44h28m47.505100086s) [upload=39171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 513.722158ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
2.912748259s
FAIL (2025-08-31 11:18:14 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.912774048s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
862.003864ms
FAIL (2025-08-31 11:18:09 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 862.02886ms
core
uplink
eu1
|
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)antiblip:
4 out of 15 is failed (threshold=6) [.X.......XX..X.] duration:
83.305911ms remote:
"tcp:34.202.203.47:443" response headers:
Cf-Cache-Status=HIT Age=403947 Last-Modified=Tue, 26 Aug 2025 19:05:15 GMT Surrogate-Control=max-age=2147483647 Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 Content-Type=text/html Cf-Ray=977c1f7f5d28f30f-IAD Surrogate-Key=www.storj.io 65d78d3b8dbb1f54a94020c7 pageId:67eb0cf63d37c62ac04e0ff0 X-Lambda-Id=c6207948-cc73-40b2-9a2a-e948767f3d5e Set-Cookie=_cfuvid=k51S_I0dcpJpqsavOOGE57wUF93zJNGveMi.Xd.kRaE-1756639062976-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Date=Sun, 31 Aug 2025 11:17:42 GMT X-Cluster-Name=us-east-1-prod-hosting-red target:
https://www.storj.io/
FAIL (2025-08-31 11:18:05 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 83.333474ms
core
website
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
3 out of 60 is failed (threshold=3) [.XXX........................................................] duration:
47.542559ms
FAIL (2025-08-31 11:18:03 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout
Duration: 47.583617ms
edge
authservice
http
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
4.709031267s
FAIL (2025-08-31 11:18:01 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.709062785s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [XXX.........................................................] duration:
471.202257ms
FAIL (2025-08-31 11:17:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Aug 31 10:35:45.949357429 (44h27m21.302142558s) [upload=39166]: Starting upload
Aug 31 10:35:45.949453992 (44h27m21.302239119s) [upload=39166]: Done starting upload
Aug 31 10:35:45.949561915 (44h27m21.302347043s) [upload=39166]: Uploading segments...
Aug 31 10:35:45.949839758 (44h27m21.302624886s) [upload=39166]: Got next segment. Inline: false
Aug 31 10:35:45.949875034 (44h27m21.302660163s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 10:35:45.949880324 (44h27m21.302665453s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 10:35:45.974500057 (44h27m21.327285184s) [seg_pos={0 0}, upload=39166]: Begin upload segment...
Aug 31 10:35:45.974534781 (44h27m21.32731991s) [seg_pos={0 0}, upload=39166]: Done begin upload segment.
Aug 31 10:35:45.974561612 (44h27m21.32734674s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.974622567 (44h27m21.327407705s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975048299 (44h27m21.327833426s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975060932 (44h27m21.327846061s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975453572 (44h27m21.3282387s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975479350 (44h27m21.328264488s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975834078 (44h27m21.328619207s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975889633 (44h27m21.328674761s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976450218 (44h27m21.329235346s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976462792 (44h27m21.32924792s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976837658 (44h27m21.329622786s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976933107 (44h27m21.329718236s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977198717 (44h27m21.329983845s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977335244 (44h27m21.330120372s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977693609 (44h27m21.330478747s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.978139749 (44h27m21.330924877s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:46.010680273 (44h27m21.363465391s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 5.161.89.252:33308->77.90.31.17:4770: write: no route to host; manager closed: conn closed: op
Aug 31 10:35:46.094242937 (44h27m21.447028065s) [upload=39166]: Next returned nil segment
Aug 31 10:35:46.094257233 (44h27m21.447042362s) [upload=39166]: Waiting for error group managing segments...
Aug 31 10:35:46.107643376 (44h27m21.460428494s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.108127397 (44h27m21.460912515s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.109366731 (44h27m21.462151858s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.158408011 (44h27m21.51119313s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.160289913 (44h27m21.513075032s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.175403087 (44h27m21.528188235s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.180837789 (44h27m21.533622927s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.191657048 (44h27m21.544442177s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.201182052 (44h27m21.55396718s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.203261847 (44h27m21.556046995s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:36:15.930363208 (44h27m51.283148347s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.930627486 (44h27m51.283412614s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op
Aug 31 10:36:15.930934504 (44h27m51.283719632s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 5.161.89.252:18584->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931120965 (44h27m51.283906103s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931311063 (44h27m51.284096192s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 5.161.89.252:48956->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931413836 (44h27m51.284198964s) [seg_pos={0 0}, upload=39166]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Aug 31 10:36:15.954490543 (44h27m51.307275682s) [upload=39166]: Done uploading segments...
Aug 31 10:36:15.954518606 (44h27m51.307303744s) [upload=39166]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 471.233475ms
core
uplink
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
2 out of 60 is failed (threshold=3) [...XX.......................................................] duration:
144.330161ms
FAIL (2025-08-31 11:17:54 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp: lookup auth.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:62187->100.100.100.100:53: i/o timeout
Duration: 144.349927ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
130.130169ms
FAIL (2025-08-31 11:17:52 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 130.15759ms
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
2 out of 15 is failed (threshold=6) [....X.........X] duration:
59.629868ms remote:
"tcp:54.243.86.28:443" response headers:
Cache-Control=private Alt-Svc=h3=":443"; ma=86400 X-Cluster-Name=us-east-1-prod-hosting-red Date=Sun, 31 Aug 2025 11:17:50 GMT Content-Length=166 Cf-Ray=977c1fb069c3d697-IAD Cf-Cache-Status=BYPASS Vary=Accept-Encoding Set-Cookie=_cfuvid=Dmeli_aGxBKFgSt8S3gcL6iZnzp0F43lPBd5BQ_WGoU-1756639070832-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Content-Type=text/html Location=https://static.storj.io/dcs-satellites target:
https://www.storj.io/dcs-satellites
FAIL (2025-08-31 11:17:50 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 59.64709ms
core
website
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
164.228323ms target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-31 11:17:45 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout
Duration: 164.250166ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)antiblip:
5 out of 15 is failed (threshold=6) [X.X.......XX..X] duration:
61.083612ms remote:
"tcp:34.202.203.47:443" response headers:
Cf-Cache-Status=HIT Age=403947 Last-Modified=Tue, 26 Aug 2025 19:05:15 GMT Surrogate-Control=max-age=2147483647 Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 Content-Type=text/html Cf-Ray=977c1f7f5d28f30f-IAD Surrogate-Key=www.storj.io 65d78d3b8dbb1f54a94020c7 pageId:67eb0cf63d37c62ac04e0ff0 X-Lambda-Id=c6207948-cc73-40b2-9a2a-e948767f3d5e Set-Cookie=_cfuvid=k51S_I0dcpJpqsavOOGE57wUF93zJNGveMi.Xd.kRaE-1756639062976-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Date=Sun, 31 Aug 2025 11:17:42 GMT X-Cluster-Name=us-east-1-prod-hosting-red target:
https://www.storj.io/
FAIL (2025-08-31 11:17:42 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 61.106605ms
core
website
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.038576782s
FAIL (2025-08-31 11:17:40 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.038623561s
core
uplink
ap1
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration:
59.833019ms
FAIL (2025-08-31 11:17:35 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout
Duration: 59.859217ms
edge
authservice
http
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [XXXX........................................................] duration:
578.91021ms
FAIL (2025-08-31 11:17:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Aug 31 10:35:45.949357429 (44h27m21.302142558s) [upload=39166]: Starting upload
Aug 31 10:35:45.949453992 (44h27m21.302239119s) [upload=39166]: Done starting upload
Aug 31 10:35:45.949561915 (44h27m21.302347043s) [upload=39166]: Uploading segments...
Aug 31 10:35:45.949839758 (44h27m21.302624886s) [upload=39166]: Got next segment. Inline: false
Aug 31 10:35:45.949875034 (44h27m21.302660163s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 10:35:45.949880324 (44h27m21.302665453s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 10:35:45.974500057 (44h27m21.327285184s) [seg_pos={0 0}, upload=39166]: Begin upload segment...
Aug 31 10:35:45.974534781 (44h27m21.32731991s) [seg_pos={0 0}, upload=39166]: Done begin upload segment.
Aug 31 10:35:45.974561612 (44h27m21.32734674s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.974622567 (44h27m21.327407705s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975048299 (44h27m21.327833426s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975060932 (44h27m21.327846061s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975453572 (44h27m21.3282387s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975479350 (44h27m21.328264488s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975834078 (44h27m21.328619207s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975889633 (44h27m21.328674761s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976450218 (44h27m21.329235346s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976462792 (44h27m21.32924792s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976837658 (44h27m21.329622786s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976933107 (44h27m21.329718236s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977198717 (44h27m21.329983845s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977335244 (44h27m21.330120372s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977693609 (44h27m21.330478747s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.978139749 (44h27m21.330924877s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:46.010680273 (44h27m21.363465391s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 5.161.89.252:33308->77.90.31.17:4770: write: no route to host; manager closed: conn closed: op
Aug 31 10:35:46.094242937 (44h27m21.447028065s) [upload=39166]: Next returned nil segment
Aug 31 10:35:46.094257233 (44h27m21.447042362s) [upload=39166]: Waiting for error group managing segments...
Aug 31 10:35:46.107643376 (44h27m21.460428494s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.108127397 (44h27m21.460912515s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.109366731 (44h27m21.462151858s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.158408011 (44h27m21.51119313s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.160289913 (44h27m21.513075032s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.175403087 (44h27m21.528188235s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.180837789 (44h27m21.533622927s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.191657048 (44h27m21.544442177s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.201182052 (44h27m21.55396718s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.203261847 (44h27m21.556046995s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:36:15.930363208 (44h27m51.283148347s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.930627486 (44h27m51.283412614s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op
Aug 31 10:36:15.930934504 (44h27m51.283719632s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 5.161.89.252:18584->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931120965 (44h27m51.283906103s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931311063 (44h27m51.284096192s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 5.161.89.252:48956->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931413836 (44h27m51.284198964s) [seg_pos={0 0}, upload=39166]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Aug 31 10:36:15.954490543 (44h27m51.307275682s) [upload=39166]: Done uploading segments...
Aug 31 10:36:15.954518606 (44h27m51.307303744s) [upload=39166]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 578.932782ms
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [.....X.........] duration:
96.665402ms remote:
"tcp:34.202.203.47:443" response headers:
Date=Sun, 31 Aug 2025 11:10:53 GMT Content-Type=text/html Content-Length=166 Cf-Ray=977c157d4ec1ef65-IAD Cf-Cache-Status=BYPASS Cache-Control=private Set-Cookie=_cfuvid=H9OgB3lYDB5f2QaWWAx6oBKC3Vwn8mpQ8Ru5rkqPJ3A-1756638653055-0.0.1.1-604800000; path=/; domain=.cdn.webflow.com; HttpOnly; Secure; SameSite=None Location=https://static.storj.io/dcs-satellites Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 X-Cluster-Name=us-east-1-prod-hosting-red target:
https://www.storj.io/dcs-satellites
FAIL (2025-08-31 11:17:29 UTC)
orbiter warning: no valid certificate chain appears to exist after 336h0m0s
Duration: 96.696342ms
core
website
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..XX........................................................] duration:
2.928386222s
FAIL (2025-08-31 11:17:27 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.928414846s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.251759469s
FAIL (2025-08-31 11:17:27 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 2.251787293s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
2 out of 60 is failed (threshold=3) [....XX......................................................] duration:
152.867406ms
FAIL (2025-08-31 11:17:21 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp: lookup auth.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:62187->100.100.100.100:53: i/o timeout
Duration: 152.888074ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [XX.XXX......................................................] duration:
505.04343ms
FAIL (2025-08-31 11:17:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
g 31 10:36:12.150915169 (44h27m47.503700297s) [upload=39171]: Done starting upload
Aug 31 10:36:12.151058448 (44h27m47.503843586s) [upload=39171]: Uploading segments...
Aug 31 10:36:12.151350147 (44h27m47.504135276s) [upload=39171]: Got next segment. Inline: false
Aug 31 10:36:12.151407877 (44h27m47.504193015s) [upload=39171]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 10:36:12.151414058 (44h27m47.504199186s) [upload=39171]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 10:36:12.181724654 (44h27m47.534509782s) [seg_pos={0 0}, upload=39171]: Begin upload segment...
Aug 31 10:36:12.181759048 (44h27m47.534544176s) [seg_pos={0 0}, upload=39171]: Done begin upload segment.
Aug 31 10:36:12.181787412 (44h27m47.534572551s) [noise=true , address=104.36.82.47:4543 , piece_id=OKISSUYRQP4GGIRDDPYFO6MCHDAFJEWCGHLI3P5T7EHBGOD7CYQA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.181865228 (44h27m47.534650367s) [noise=true , address=45.91.116.12:4239 , piece_id=JIN2FAKX4JF3WVYMMICMIFQDIJT7ZZ4XCVCTDTKB7B3OFDTBY6HQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182335613 (44h27m47.535120752s) [noise=true , address=77.90.31.10:4545 , piece_id=R5HG7NAZ2BFJCIYJOMV4BAD3Q7MAANXZJIFUDHBJ56PMITAB7FJA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182439810 (44h27m47.535224938s) [noise=true , address=104.36.82.11:4130 , piece_id=NEJRZC6MR7DTWS62XZZPJXTZW6GVHK4HIK6BXFD3QI46UGQACR3Q, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182733102 (44h27m47.53551823s) [noise=true , address=45.91.116.7:4034 , piece_id=Z5BA6X346T3H2CITKWTLJDZOUOTVCHTXPUDPO2OKKXGRQF3YUMKQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.182828772 (44h27m47.535613901s) [noise=true , address=79.127.205.246:4280 , piece_id=BG3RNCBTBDNRY3MMJUQNKJP2RWH57UODQBKK66KK2WEDXA6FJQWA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183134137 (44h27m47.535919276s) [noise=true , address=104.36.82.48:4589 , piece_id=U4VSTDANJ6ZGA6PTOLZBKGBLD6K73SG4FQT6PUIH3QPWIO5A7YQQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183571981 (44h27m47.53635711s) [noise=true , address=77.90.31.13:4629 , piece_id=746YIFKZ7XTJCLHAV2K2TWYRIJP5PFS47B5D37PFT6MERB62QM6Q, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.183905159 (44h27m47.536690288s) [noise=true , address=104.36.82.55:4212 , piece_id=BBTOTIURLHZXSY3NUBOJXHOB4U3HJIFEJM2AUO35DFACNAWBEZJA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184080729 (44h27m47.536865858s) [noise=true , address=45.91.116.13:4176 , piece_id=TTOEVZ6NGPERE27QNANZQVNP2QSCVGITHW5EFOV34EJ2KJ2GTVQA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184383600 (44h27m47.537168737s) [noise=true , address=77.90.31.17:4787 , piece_id=YLZWY3HQRSKHUHBOOVY36TATZ5IN7X2C27IXXCJKZZF4MHSRYH4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184492785 (44h27m47.537277914s) [noise=true , address=77.90.31.14:4678 , piece_id=4XGVSFNNY6MGP5IOHJIBXL3S7JIPYAXKHFIAE6BSX5TAKTCJ4JVQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184744750 (44h27m47.537529878s) [noise=true , address=45.91.116.9:4281 , piece_id=EWWXWQPBEDE724UFU2Z7YJ4E5DUI45OMNLHEFCC4OMG7ZQLWWBBQ, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.184850529 (44h27m47.537635657s) [noise=true , address=104.36.82.56:4264 , piece_id=S67WK5JPS46KPUHBQPLDDRGYJOLN6EEVOE5BFYT74U2IIXZFWXTA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185105669 (44h27m47.537890807s) [noise=true , address=77.90.31.21:4891 , piece_id=IKWWTL4THNTJGQ3CQLADO4KY2KFOXPASHSALNGE7NBUAXHPV3MBA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185259740 (44h27m47.538044868s) [noise=true , address=64.49.13.138:4482 , piece_id=JNYXEQ6PTZYSW5ULU55W5ZFEBEW25KXCDA36K47K4ADKODUZOKYA, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185545127 (44h27m47.538330255s) [noise=true , address=104.36.82.23:4458 , piece_id=Y5IELCEOPOLHYYU2WEAZM7ZQMS3W6NR5YS64Y4UYD5MYZZQUJY4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.185657418 (44h27m47.538442556s) [noise=true , address=104.36.82.44:4883 , piece_id=455DKBRQXXYON4JM5CQ4SE5WR3WTTH3AJUAB6ZEHEHSE4RE6LP4A, seg_pos={0 0}, upload=39171]: Uploading piece...
Aug 31 10:36:12.220474138 (44h27m47.573259266s) [noise=true , address=77.90.31.21:4891 , piece_id=IKWWTL4THNTJGQ3CQLADO4KY2KFOXPASHSALNGE7NBUAXHPV3MBA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: write tcp 5.161.89.252:27554->77.90.31.21:4891: write: no route to host; conn closed: op
Aug 31 10:36:12.302528983 (44h27m47.655314111s) [upload=39171]: Next returned nil segment
Aug 31 10:36:12.302541757 (44h27m47.655326885s) [upload=39171]: Waiting for error group managing segments...
Aug 31 10:36:12.316638798 (44h27m47.669423926s) [noise=true , address=104.36.82.56:4264 , piece_id=S67WK5JPS46KPUHBQPLDDRGYJOLN6EEVOE5BFYT74U2IIXZFWXTA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.316773852 (44h27m47.66955899s) [noise=true , address=104.36.82.44:4883 , piece_id=455DKBRQXXYON4JM5CQ4SE5WR3WTTH3AJUAB6ZEHEHSE4RE6LP4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317388379 (44h27m47.670173506s) [noise=true , address=104.36.82.47:4543 , piece_id=OKISSUYRQP4GGIRDDPYFO6MCHDAFJEWCGHLI3P5T7EHBGOD7CYQA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317831102 (44h27m47.67061623s) [noise=true , address=104.36.82.23:4458 , piece_id=Y5IELCEOPOLHYYU2WEAZM7ZQMS3W6NR5YS64Y4UYD5MYZZQUJY4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.317981115 (44h27m47.670766243s) [noise=true , address=104.36.82.55:4212 , piece_id=BBTOTIURLHZXSY3NUBOJXHOB4U3HJIFEJM2AUO35DFACNAWBEZJA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.318644403 (44h27m47.671429531s) [noise=true , address=104.36.82.11:4130 , piece_id=NEJRZC6MR7DTWS62XZZPJXTZW6GVHK4HIK6BXFD3QI46UGQACR3Q, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.319463665 (44h27m47.672248794s) [noise=true , address=104.36.82.48:4589 , piece_id=U4VSTDANJ6ZGA6PTOLZBKGBLD6K73SG4FQT6PUIH3QPWIO5A7YQQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.365155230 (44h27m47.717940358s) [noise=true , address=45.91.116.13:4176 , piece_id=TTOEVZ6NGPERE27QNANZQVNP2QSCVGITHW5EFOV34EJ2KJ2GTVQA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.382100673 (44h27m47.734885801s) [noise=true , address=45.91.116.9:4281 , piece_id=EWWXWQPBEDE724UFU2Z7YJ4E5DUI45OMNLHEFCC4OMG7ZQLWWBBQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.382477341 (44h27m47.73526248s) [noise=true , address=45.91.116.12:4239 , piece_id=JIN2FAKX4JF3WVYMMICMIFQDIJT7ZZ4XCVCTDTKB7B3OFDTBY6HQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.385851305 (44h27m47.738636442s) [noise=true , address=45.91.116.7:4034 , piece_id=Z5BA6X346T3H2CITKWTLJDZOUOTVCHTXPUDPO2OKKXGRQF3YUMKQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.670547674 (44h27m48.023332772s) [noise=true , address=79.127.205.246:4280 , piece_id=BG3RNCBTBDNRY3MMJUQNKJP2RWH57UODQBKK66KK2WEDXA6FJQWA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:36:12.734232925 (44h27m48.087018034s) [noise=true , address=64.49.13.138:4482 , piece_id=JNYXEQ6PTZYSW5ULU55W5ZFEBEW25KXCDA36K47K4ADKODUZOKYA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: <nil>
Aug 31 10:37:12.136109118 (44h28m47.488894215s) [noise=true , address=77.90.31.17:4787 , piece_id=YLZWY3HQRSKHUHBOOVY36TATZ5IN7X2C27IXXCJKZZF4MHSRYH4A, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: rpc: dial tcp 77.90.31.17:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136180632 (44h28m47.48896573s) [noise=true , address=77.90.31.10:4545 , piece_id=R5HG7NAZ2BFJCIYJOMV4BAD3Q7MAANXZJIFUDHBJ56PMITAB7FJA, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:1ZFFijCUFET1ShEJuav6uAzFhfraCEcr6seuySYN9ipn4wsZGX, address:77.90.31.10:4545): protocol: rpc: dial tcp 77.90.31.10:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136217150 (44h28m47.489002248s) [noise=true , address=77.90.31.13:4629 , piece_id=746YIFKZ7XTJCLHAV2K2TWYRIJP5PFS47B5D37PFT6MERB62QM6Q, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136244112 (44h28m47.48902921s) [noise=true , address=77.90.31.14:4678 , piece_id=4XGVSFNNY6MGP5IOHJIBXL3S7JIPYAXKHFIAE6BSX5TAKTCJ4JVQ, seg_pos={0 0}, upload=39171]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:37:12.136292463 (44h28m47.48907756s) [seg_pos={0 0}, upload=39171]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Aug 31 10:37:12.152297525 (44h28m47.505082633s) [upload=39171]: Done uploading segments...
Aug 31 10:37:12.152314989 (44h28m47.505100086s) [upload=39171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 505.09081ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
880.333965ms
FAIL (2025-08-31 11:17:12 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 880.358531ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
219.601184ms
FAIL (2025-08-31 11:17:04 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 219.619678ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
2.061362091s
FAIL (2025-08-31 11:17:02 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.061389413s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
130.835958ms
FAIL (2025-08-31 11:16:53 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 130.855475ms
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
50.249013ms target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-31 11:16:52 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp: lookup uptime-checks.gateway.storjshare.io on 100.100.100.100:53: read udp 100.64.0.137:29236->100.100.100.100:53: i/o timeout
Duration: 50.26874ms
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration:
471.079307ms
FAIL (2025-08-31 11:16:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Aug 31 10:35:45.949357429 (44h27m21.302142558s) [upload=39166]: Starting upload
Aug 31 10:35:45.949453992 (44h27m21.302239119s) [upload=39166]: Done starting upload
Aug 31 10:35:45.949561915 (44h27m21.302347043s) [upload=39166]: Uploading segments...
Aug 31 10:35:45.949839758 (44h27m21.302624886s) [upload=39166]: Got next segment. Inline: false
Aug 31 10:35:45.949875034 (44h27m21.302660163s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 10:35:45.949880324 (44h27m21.302665453s) [upload=39166]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 10:35:45.974500057 (44h27m21.327285184s) [seg_pos={0 0}, upload=39166]: Begin upload segment...
Aug 31 10:35:45.974534781 (44h27m21.32731991s) [seg_pos={0 0}, upload=39166]: Done begin upload segment.
Aug 31 10:35:45.974561612 (44h27m21.32734674s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.974622567 (44h27m21.327407705s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975048299 (44h27m21.327833426s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975060932 (44h27m21.327846061s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975453572 (44h27m21.3282387s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975479350 (44h27m21.328264488s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975834078 (44h27m21.328619207s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.975889633 (44h27m21.328674761s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976450218 (44h27m21.329235346s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976462792 (44h27m21.32924792s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976837658 (44h27m21.329622786s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.976933107 (44h27m21.329718236s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977198717 (44h27m21.329983845s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977335244 (44h27m21.330120372s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.977693609 (44h27m21.330478747s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:45.978139749 (44h27m21.330924877s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Uploading piece...
Aug 31 10:35:46.010680273 (44h27m21.363465391s) [noise=true , address=77.90.31.17:4770 , piece_id=GMBH47VTWWVOFSYF5HSN2FTTQZS6MUYJZ2C3243INM4TKE3F4ZJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 5.161.89.252:33308->77.90.31.17:4770: write: no route to host; manager closed: conn closed: op
Aug 31 10:35:46.094242937 (44h27m21.447028065s) [upload=39166]: Next returned nil segment
Aug 31 10:35:46.094257233 (44h27m21.447042362s) [upload=39166]: Waiting for error group managing segments...
Aug 31 10:35:46.107643376 (44h27m21.460428494s) [noise=true , address=104.36.82.15:4259 , piece_id=7N4OLUFMAZPMGDZCBPQWA5RI3QU4KTD4K53FXRBSAA6RDBKZLBVA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.108127397 (44h27m21.460912515s) [noise=true , address=104.36.82.2:4730 , piece_id=NMRNV2ZOG7JDLKSYHNAG7FFLUR7MECSBEZK57FVZ2FKJCX6BHM2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.109366731 (44h27m21.462151858s) [noise=true , address=104.36.82.5:4840 , piece_id=VOFINL532TIV77L5WVMMLOU6ABJBDYYUNX7U4XSBSFJE3JQICJQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.158408011 (44h27m21.51119313s) [noise=true , address=45.91.116.11:4117 , piece_id=Q4W2NO374GGMKEUS4HFVH2U2PPRCBS3BAUKKKU2KEESFQNVB5F6Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.160289913 (44h27m21.513075032s) [noise=true , address=45.91.116.9:4262 , piece_id=2DTFYRZDKYB73XUA2QXKATIABE522EIFS2V5IGT3DPY6227F4Q4A, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.175403087 (44h27m21.528188235s) [noise=true , address=45.91.116.4:4940 , piece_id=2LEW4ZRRSCYGNX2MZCUGZJPUSNSD7ZHI63WM7FNIODHGP5K5J2HA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.180837789 (44h27m21.533622927s) [noise=true , address=45.91.116.7:4010 , piece_id=P76FQJ7UMNJZZOVAED6OYJ4KTB67KHOZYJ7X4KYQLMZAVDYHPY2Q, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.191657048 (44h27m21.544442177s) [noise=true , address=45.91.116.8:4047 , piece_id=PRK5W24CWG4TY3Q5D4YUPCKXXQ5KDGMI25L3DZ6Z3SPWIPV2OGZA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.201182052 (44h27m21.55396718s) [noise=true , address=45.91.116.10:4081 , piece_id=2UPUNDEYUJXFD44M62CXSKFNFGTVY2F5UKYZOVXHMZKIHTFB6DUQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:35:46.203261847 (44h27m21.556046995s) [noise=true , address=45.91.116.6:4973 , piece_id=VGG3ORU6E2PDMAGENBR2AJWRMQ22QLAFI5DAQUB4XMO4KI5G4TQQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: <nil>
Aug 31 10:36:15.930363208 (44h27m51.283148347s) [noise=true , address=77.90.31.17:4772 , piece_id=EXYIUPQYWHMWJX6YDTK7NMANOLDI6UPQK3SYUKPH4N42MED2CWYA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.930627486 (44h27m51.283412614s) [noise=true , address=77.90.31.16:4735 , piece_id=VT2UI46533CVPZLAMRC5TNZKQCCUWUWRBFXUUDZH7NWYVBYQXBJA, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op
Aug 31 10:36:15.930934504 (44h27m51.283719632s) [noise=true , address=77.90.31.10:4517 , piece_id=ETZJZFDL3HNZA3GSJS4GE2QZZCLSVYBZEDD2CWZIARBQ7XBVKRHQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 5.161.89.252:18584->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931120965 (44h27m51.283906103s) [noise=true , address=77.90.31.14:4662 , piece_id=5GG7POTFX2VDBHTSDLLRARV5OQ5SNWUXASAFC734IGNLJIOR34EQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931311063 (44h27m51.284096192s) [noise=true , address=77.90.31.12:4590 , piece_id=ROSYARSOOVQHYE7VI4QQFXVTRWHXMZCVR2JS4WG2PTUBSXJDANGQ, seg_pos={0 0}, upload=39166]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 5.161.89.252:48956->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 10:36:15.931413836 (44h27m51.284198964s) [seg_pos={0 0}, upload=39166]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Aug 31 10:36:15.954490543 (44h27m51.307275682s) [upload=39166]: Done uploading segments...
Aug 31 10:36:15.954518606 (44h27m51.307303744s) [upload=39166]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 471.126965ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
4.785056626s
FAIL (2025-08-31 11:16:49 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.785083307s
core
uplink
ap1
|
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
9.354893ms target:
http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt
FAIL (2025-08-31 11:16:46 UTC)
orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout
Duration: 9.371394ms
edge
linksharing
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..XX........................................................] duration:
953.13736ms
FAIL (2025-08-31 11:16:40 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 953.164973ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
2.178493556s
FAIL (2025-08-31 11:16:38 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 2.178528832s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
29.263676ms target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-31 11:16:37 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout
Duration: 29.363553ms
edge
gateway
vhost
us-select-1
|