Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..........................X.............X...................] duration:
6.000029348s trace:
https://jaeger.dev.storj.io/trace/508ccd1bd1c790cc
FAIL (2026-01-28 13:30:36 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 6.000053614s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................X.................................] duration:
476.492511ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:30:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 476.509862ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
2.271124965s trace:
https://jaeger.dev.storj.io/trace/2602f4ec6aa4253
FAIL (2026-01-28 13:30:06 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 2.271148908s
core
uplink
ap1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X....X.........................] duration:
331.572892ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:29:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 331.592547ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................X................................] duration:
451.153679ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:29:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 451.223279ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
13 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XX..............................................] duration:
2.247999078s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:29:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.248020808s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [...........................X.............X..................] duration:
4.544498312s trace:
https://jaeger.dev.storj.io/trace/508ccd1bd1c790cc
FAIL (2026-01-28 13:29:42 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.544525071s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [............................X.............X.................] duration:
4.692497615s trace:
https://jaeger.dev.storj.io/trace/508ccd1bd1c790cc
FAIL (2026-01-28 13:29:24 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.692520398s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
14 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XX.............................................] duration:
2.721139007s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:29:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.721159997s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X....X........................] duration:
368.529316ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:29:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 368.546118ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [............................X...............................] duration:
538.370339ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:29:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 538.38687ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
2.153318074s trace:
https://jaeger.dev.storj.io/trace/2602f4ec6aa4253
FAIL (2026-01-28 13:28:58 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 2.153336678s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.............................X.............X................] duration:
5.559779387s trace:
https://jaeger.dev.storj.io/trace/508ccd1bd1c790cc
FAIL (2026-01-28 13:28:50 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 5.559830985s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............................X..............................] duration:
498.227203ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:28:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 498.249294ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
15 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XX............................................] duration:
2.774307583s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:28:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.774329614s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
2.663965488s trace:
https://jaeger.dev.storj.io/trace/2602f4ec6aa4253
FAIL (2026-01-28 13:28:05 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 2.663991708s
core
uplink
ap1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X....X.......................] duration:
339.640463ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:28:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 339.658547ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..............................X.............X...............] duration:
8.940539446s trace:
https://jaeger.dev.storj.io/trace/508ccd1bd1c790cc
FAIL (2026-01-28 13:27:51 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 8.940561727s
core
uplink
ap1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................X....X......................] duration:
354.136775ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:27:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 354.152564ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
16 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XX...........................................] duration:
2.680877776s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:27:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.680904656s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.339965507s trace:
https://jaeger.dev.storj.io/trace/2602f4ec6aa4253
FAIL (2026-01-28 13:27:17 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 2.339982368s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............................X.............................] duration:
457.591064ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:27:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 457.609479ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X....X.....................] duration:
331.690625ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:27:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 331.70923ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [...............................X.............X..............] duration:
4.804061382s trace:
https://jaeger.dev.storj.io/trace/508ccd1bd1c790cc
FAIL (2026-01-28 13:26:56 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.804080839s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XX..........................................] duration:
2.730759685s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:26:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.730781768s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............................X............................] duration:
580.754454ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:26:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 580.776232ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [................................X.............X.............] duration:
4.750593104s trace:
https://jaeger.dev.storj.io/trace/508ccd1bd1c790cc
FAIL (2026-01-28 13:26:26 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.750638708s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
3.545880194s trace:
https://jaeger.dev.storj.io/trace/2602f4ec6aa4253
FAIL (2026-01-28 13:26:13 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 3.545902325s
core
uplink
ap1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................X....X....................] duration:
344.329412ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:26:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 344.348467ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.................................X.............X............] duration:
4.496340122s trace:
https://jaeger.dev.storj.io/trace/508ccd1bd1c790cc
FAIL (2026-01-28 13:25:34 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.496370048s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................................X...........................] duration:
462.013649ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:25:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 462.03544ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XX.........................................] duration:
2.728560246s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:25:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.728577398s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
2.897440722s trace:
https://jaeger.dev.storj.io/trace/2602f4ec6aa4253
FAIL (2026-01-28 13:25:14 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 2.897459437s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................................X..........................] duration:
518.315097ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:25:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 518.337809ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X....X...................] duration:
343.216704ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:24:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 343.236189ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XX........................................] duration:
2.201183127s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:24:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.201205749s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
2.34805169s trace:
https://jaeger.dev.storj.io/trace/2602f4ec6aa4253
FAIL (2026-01-28 13:24:53 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 2.34807881s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................................X.........................] duration:
471.669347ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:24:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 471.685197ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
20 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XX.......................................] duration:
2.255724182s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:24:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.255740511s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X....X..................] duration:
334.103299ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:24:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 334.124179ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..................................X.............X...........] duration:
4.632067404s trace:
https://jaeger.dev.storj.io/trace/508ccd1bd1c790cc
FAIL (2026-01-28 13:24:23 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 4.632084687s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................................X........................] duration:
523.823435ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:24:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 523.845155ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
2.656710759s trace:
https://jaeger.dev.storj.io/trace/2602f4ec6aa4253
FAIL (2026-01-28 13:24:00 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 2.656753991s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [...................................X.............X..........] duration:
8.278727617s trace:
https://jaeger.dev.storj.io/trace/508ccd1bd1c790cc
FAIL (2026-01-28 13:23:59 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 8.278752193s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
2.221700925s trace:
https://jaeger.dev.storj.io/trace/2602f4ec6aa4253
FAIL (2026-01-28 13:23:43 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 2.221720421s
core
uplink
ap1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X....X.................] duration:
378.477822ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:23:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 378.500123ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
20 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXX.XX......................................] duration:
2.71235733s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:23:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.712377668s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
2.154208006s trace:
https://jaeger.dev.storj.io/trace/2602f4ec6aa4253
FAIL (2026-01-28 13:23:10 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 2.154230259s
core
uplink
ap1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................X....X................] duration:
341.157995ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:22:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 341.175388ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................................X.......................] duration:
444.570202ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:22:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 444.590871ms
core
uplink
us-select-1
us1-placement-20
|